3D model hairstyle processing method and device, and electronic device

Abstract

The application provides a 3D model hairstyle processing method and device and electronic equipment. The matching point pairs on the hairstyle mask image of the model plane rendering image of the reference image and the 3D model to be processed are determined, and the change constraint condition is determined according to the relative position relationship of the matching point pairs to adjust the hairstyle part of the 3D model to be processed, so that the hairstyle area of the 3D model to be processed approaches the reference image. In this way, the adjustment of the hairstyle area of the 3D model to be processed can be automatically realized only by providing the 2D reference image by the user, and the diversified generation requirement of the hairstyle area of the 3D model is met.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to a method, apparatus, and electronic device for processing 3D model hairstyles. Background Technology

[0002] With the continuous development of online social technologies, the application of digital virtual avatars is becoming increasingly widespread, and the demand for diverse and personalized virtual avatars is also increasing. In traditional virtual avatar generation solutions, users can only choose preset hairstyles for the hairstyle area, which is difficult to meet the massive and diverse needs for virtual avatar generation; alternatively, users can manually adjust the hairstyle based on the preset model, but this requires cumbersome operations. Summary of the Invention

[0003] To overcome the aforementioned shortcomings in the prior art, the purpose of this application is to provide a 3D model hairstyle processing method, the method comprising:

[0004] Acquire a reference image and a 3D model to be processed, and acquire a planar rendered image of the 3D model to be processed from a preset viewpoint;

[0005] The model planar rendering image and the reference image are respectively input into the image semantic segmentation model to obtain a first hairstyle mask image indicating the hairstyle region in the model planar rendering image and a second hairstyle mask image indicating the hairstyle region in the reference image;

[0006] A dynamic programming algorithm is used to determine a first matching point pair on the first hairstyle mask image and the second hairstyle mask image. The first matching point pair includes a first hairstyle planar contour point on the first hairstyle mask image and a second hairstyle planar contour point on the second hairstyle mask image.

[0007] Determine the first hairstyle planar contour point and the second hairstyle planar contour point corresponding to the first hairstyle spatial contour point and the second hairstyle spatial contour point in the space where the 3D model to be processed is located.

[0008] The variation constraints are determined based on the positional relationships between the first hair planar contour points, the second hair planar contour points, the first hair spatial contour points, and the points on the 3D model to be processed.

[0009] The points in the hairstyle section of the 3D model to be processed are adjusted according to the changing constraints.

[0010] In one possible implementation, before the step of inputting the model planar rendered image and the reference image into the image semantic segmentation model respectively, the method further includes:

[0011] Facial landmark alignment adjustment processing is performed on the model planar rendered image and the reference image.

[0012] In one possible implementation, the step of acquiring the reference image and the 3D model to be processed includes:

[0013] Acquire a reference image and identify the hairstyle category of the reference image;

[0014] The corresponding 3D model to be processed is determined based on the recognition result of the hairstyle category identification.

[0015] In one possible implementation, the hairstyle portion of the 3D model to be processed includes multiple facets; prior to the step of determining the variation constraints based on the positional relationship between the first and second hairstyle spatial contour points and the positional relationship between points on the 3D model to be processed, the method further includes:

[0016] Remove duplicate points formed by the overlapping of different facets in the hairstyle part of the 3D model to be processed;

[0017] And / or, add connection constraint patches between different patches in the hairstyle section of the 3D model to be processed.

[0018] In one possible implementation, the variation constraints include a first constraint and a second constraint; the step of determining the variation constraints based on the positional relationship between the first and second hairstyle space contour points and the positional relationship between points on the 3D model to be processed includes:

[0019] Based on the positional relationship between the first hair planar contour point and the second hair planar contour point and the positional relationship between the first hair spatial contour point and the second hair spatial contour point, the target spatial contour point after performing thin-plate spline interpolation transformation on the first hair spatial contour point is determined, with the first hair spatial contour point being as close as possible to the target spatial contour point as the first constraint condition.

[0020] The second constraint is to maximize the number of points that maintain the connection relationship unchanged, based on the connection relationship between adjacent points in the 3D model to be processed.

[0021] In one possible implementation, the 3D model to be processed includes a head portion and a hairstyle portion; the constraints further include a third constraint; the step of determining the changing constraint based on the positional relationship between the first hairstyle spatial contour point and the second hairstyle spatial contour point and the positional relationship between points on the 3D model to be processed further includes:

[0022] The third constraint condition is to determine multiple points in the hairstyle section that are within a preset range from the head section as fixed points, so as to keep the position of the fixed points unchanged.

[0023] In one possible implementation, the step of determining the target spatial contour point after performing thin-plate spline interpolation transformation on the first hairstyle spatial contour point based on the positional relationship between the first hairstyle planar contour point and the second hairstyle planar contour point and the positional relationship between the first hairstyle spatial contour point and the second hairstyle spatial contour point includes:

[0024] From the points in the 3D model to be processed that are not visible within the preset viewpoint, determine the points that are within a set range of the first hairstyle space contour points as the third hairstyle space contour points.

[0025] Determine the first positional relationship between the first hairline planar contour point and the second hairline planar contour point, and determine the second positional relationship between the first hairline spatial contour point, the third hairline spatial contour point, and the second hairline spatial contour point;

[0026] Based on the first positional relationship and the second positional relationship, determine the target spatial contour point after performing thin-plate spline interpolation transformation on the first hairstyle spatial contour point.

[0027] Another object of this application is to provide a 3D model hairstyle processing device, the 3D model hairstyle processing device comprising:

[0028] The image acquisition module is used to acquire a reference image and a 3D model to be processed, and to acquire a model plane rendering image of the 3D model to be processed under a preset viewpoint.

[0029] The semantic segmentation module is used to input the model planar rendering image and the reference image into the image semantic segmentation model respectively to obtain a first hairstyle mask image indicating the hairstyle region in the model planar rendering image and a second hairstyle mask image indicating the hairstyle region in the reference image;

[0030] The point matching module is used to determine a first matching point pair on the first hairstyle mask image and the second hairstyle mask image through a dynamic programming algorithm. The first matching point pair includes a first hairstyle planar contour point on the first hairstyle mask image and a second hairstyle planar contour point on the second hairstyle mask image.

[0031] The point mapping module is used to determine the first hairstyle plane contour point and the second hairstyle plane contour point corresponding to the first hairstyle space contour point and the second hairstyle space contour point in the space where the 3D model to be processed is located.

[0032] The constraint determination module is used to determine the changing constraint conditions based on the positional relationship between the first hairstyle plane contour point, the second hairstyle plane contour point, the first hairstyle spatial contour point, and the points on the 3D model to be processed.

[0033] The model adjustment module is used to adjust each point in the hairstyle part of the 3D model to be processed according to the changing constraints.

[0034] Another objective of this application is to provide an electronic device, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, which, when executed by the processor, implement the 3D model hairstyle processing method provided in this application.

[0035] Another objective of this application is to provide a machine-readable storage medium, characterized in that the machine-readable storage medium stores machine-executable instructions, which, when executed by one or more processors, implement the 3D model hairstyle processing method provided in this application.

[0036] Compared with the prior art, this application has the following beneficial effects:

[0037] The 3D model hairstyle processing method, apparatus, and electronic device provided in this application determine matching point pairs on the hairstyle mask images of a reference image and a 2D rendered image of the 3D model to be processed. Based on the relative positional relationship of the matching point pairs, variation constraints are determined to adjust the hairstyle portion of the 3D model to be processed, making the hairstyle area of ​​the 3D model to be processed approximate the reference image. Thus, the adjustment of the hairstyle area of ​​the 3D model to be processed can be automatically achieved only when the user provides a 2D reference image, meeting the diverse generation needs of 3D model hairstyle areas. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating the steps of the 3D model hairstyle processing method provided in the embodiments of this application;

[0040] Figure 2 This is a schematic diagram illustrating the application scenario of the electronic device provided in the embodiments of this application;

[0041] Figure 3A schematic diagram of an electronic device provided in an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of the functional modules of the 3D model hairstyle processing device provided in the embodiments of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0044] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0045] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0046] In the description of this application, it should be noted that the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0047] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0048] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a 3D model hairstyle processing method provided in this embodiment. The steps of the method are described in detail below.

[0049] Step S110: Obtain a reference image and a 3D model to be processed, and obtain a model planar rendering image of the 3D model to be processed from a preset viewpoint.

[0050] In this embodiment, the 3D model to be processed can be a 3D model that requires adjustment of the hairstyle. The reference image can include the face and hairstyle. The reference image is used as a reference target when adjusting the 3D model to be processed.

[0051] Optionally, the outline of the 3D model varies significantly depending on the hairstyle type. For example, the outline of the 3D model differs considerably between a double ponytail hairstyle and a single ponytail hairstyle. Therefore, in this embodiment, after acquiring the reference image, hairstyle category recognition can be performed on the reference image first, and then the corresponding 3D model to be processed can be determined based on the recognition result. In this way, the hairstyle type of the 3D model to be processed can be made to be roughly consistent with the reference image, thereby reducing the amount of subsequent data processing required to adjust the 3D model and minimizing adjustment distortion.

[0052] After acquiring the reference image and the 3D model to be processed, a planar rendering image of the 3D model to be processed can be obtained from a preset viewpoint. For example, the 3D model to be processed can be rendered from the viewpoint of looking directly at the face of the 3D model to be processed to obtain the planar rendering image of the model.

[0053] Step S120: Input the model planar rendering image and the reference image into the image semantic segmentation model respectively to obtain a first hairstyle mask image indicating the hairstyle region in the model planar rendering image and a second hairstyle mask image indicating the hairstyle region in the reference image.

[0054] In this embodiment, the model's planar rendered image and the reference image can be respectively input into an image semantic segmentation model for processing. The image semantic segmentation model can be trained to identify hairstyle regions in the input image and output corresponding hairstyle mask images. For example, in the hairstyle mask image, the pixel value of the hairstyle region can be 1, and the pixel value of the non-hairstyle region can be 0. Through the image semantic segmentation model, a first hairstyle mask image indicating the hairstyle region in the model's planar rendered image and a second hairstyle mask image indicating the hairstyle region in the reference image can be obtained.

[0055] Specifically, the semantic segmentation model can employ the DeeplabV3+ model, a basic deep learning network, and can be trained to convergence using data augmentation methods. For the planar rendered image of the model, since the hairstyle of the 3D model to be processed is typically composed of numerous facets rather than fine, fragmented hair, the segmentation result is relatively stable. However, the reference image is usually a realistic portrait photograph, including numerous fragmented hairs and noise fluctuations. Therefore, the segmentation result of the reference image needs to undergo post-processing operations such as blurring, smoothing, and removing loose hairs before being processed by the next module, ultimately resulting in the second hairstyle mask image. r The calculation formula is as follows:

[0056] mask r =Pre(E seg (img-r aligned ))

[0057] Among them, img_r aligned For the reference image, E seg For semantic segmentation networks, Pre is a post-processing function for the semantic segmentation output, used to smooth edges and remove free blocks.

[0058] In some possible implementations, to improve the accuracy of subsequent processing, facial key point alignment adjustment processing can be performed on the model planar rendered image and the reference image before step S120. Specifically, the reference image can be aligned to the model planar rendered image based on key points and triangulation algorithms.

[0059] Step S130: A first matching point pair is determined on the first hairstyle mask image and the second hairstyle mask image by a dynamic programming algorithm. The first matching point pair includes a first hairstyle planar contour point on the first hairstyle mask image and a second hairstyle planar contour point on the second hairstyle mask image.

[0060] In this embodiment, to make the hairstyle of the 3D model to be processed similar to the reference image, the point changes that need to occur between the model's planar rendered image and the reference image are first determined at the 2D level, thereby constraining the point changes at the 3D level. The point changes at the 2D level require first determining the contour changes of the hairstyle in the reference image and the planar rendered image. Therefore, in this embodiment, a dynamic programming algorithm can be used to determine the first matching point pair on the first hairstyle mask image and the second hairstyle mask image. In the first matching point pair, the first hairstyle planar contour point on the first hairstyle mask image is the point that needs to be adjusted, and the second hairstyle planar contour point on the second hairstyle mask image is the target point that needs to be reached or approached.

[0061] Specifically, in this embodiment, the problem of finding matching point pairs can be transformed into an optimization problem of minimizing an energy function. Specifically, in this embodiment, boundary point sampling can be performed on the first hairstyle mask image and the second hairstyle mask image. For example, the first boundary point P of the first hairstyle mask image can be obtained by sampling using the following formula. m and the second boundary point P on the second hairstyle mask image r :

[0062] P m =Sample(FC(mask) m ))

[0063] P r =Sample(FC(mask) r ))

[0064] Among them, mask m For the first hairstyle mask image, mask r The second hairstyle mask image is defined by FC(), which is a boundary finding function, and Sample() is a function that samples boundary points.

[0065] Then the first boundary point P can be... m As the first hairline plane contour point, then at the second boundary point P r Find the point P that corresponds to the plane contour of the first hairstyle. m Matching second hairstyle planar contour points r Specifically, the second hairstyle plane contour points pofnts can be determined using the following formula. r :

[0066]

[0067] Where EP() is the energy function of a point, consisting of the Euclidean distance and the difference in normal vectors between the matching point pairs; EE() is the edge energy function of the matching points, consisting of the direction difference of the boundary formed by the matching points; by minimizing the energy function, that is, making the distance, normal vector, and edge direction between corresponding points of two matching point sets as consistent as possible, the first hairstyle plane contour point P is finally obtained. m and the corresponding second hairstyle plane contour points r .

[0068] Step S140: Determine the first hairstyle planar contour point and the second hairstyle planar contour point corresponding to the first hairstyle spatial contour point and the second hairstyle spatial contour point in the space where the 3D model to be processed is located.

[0069] In this embodiment, after determining the first matching point pair in the 2D layer, it is necessary to map the first matching point pair to the point to be processed. Figure 3 The D image is placed in 3D space to obtain a second matching point pair, which includes a first hairstyle space contour point corresponding to the first hairstyle plane contour point and a second hairstyle space contour point corresponding to the second hairstyle plane contour point.

[0070] Specifically, in this embodiment, the first hairstyle plane contour point P can be determined based on the rendering mapping matrix used when rendering the 3D model to be processed into the model plane rendering map. m Perform inverse mapping to obtain the spatial contour point P′ of the first hairstyle. m Similarly, the points representing the second hairstyle's planar contour can be obtained. r The corresponding second hairstyle space outline points′ r It is understood that, in this embodiment, the first hairstyle spatial contour point P′ m These are the contour points on the 3D model to be processed.

[0071] Step S150: Determine the variation constraints based on the positional relationships between the first hairstyle planar contour points, the second hairstyle planar contour points, the first hairstyle spatial contour points, and the points on the 3D model to be processed.

[0072] In this embodiment, some variation constraints can be determined based on the first hairline planar contour points, the second hairline planar contour points, and the first hairline spatial contour points, so that the hairline portion of the 3D model to be processed is as similar as possible to the reference image. Furthermore, to avoid excessive distortion during the adjustment process, some variation constraints also need to be determined based on the positional relationship between the first and second hairline spatial contour points and the positional relationship between points on the 3D model to be processed.

[0073] Step S160: Adjust the position of the hairstyle part of the 3D model to be processed according to the change constraint conditions.

[0074] In this embodiment, in the hairstyle part of the 3D model to be processed, in addition to the first hairstyle space contour point, there are many other contour points. These points can be changed while satisfying the change constraint conditions determined in step S 150.

[0075] Specifically, in this embodiment, non-rigid transformation constraints such as ASAP can be used to operate on the point structure. The original non-rigid transformation uses operators such as Laplace and cosine Laplace, which are for manifold structures or watertight structures. However, adding this local structural constraint to non-watertight structures can also ensure the stability of the local structure to a certain extent.

[0076] Based on the above design, the 3D model hairstyle processing method provided in this application determines matching point pairs on the hairstyle mask images of the reference image and the model planar rendering image of the 3D model to be processed, and determines variation constraints based on the relative positional relationship of the matching point pairs to adjust the hairstyle part of the 3D model to be processed, so that the hairstyle area of ​​the 3D model to be processed approaches the reference image. In this way, the adjustment of the hairstyle area of ​​the 3D model to be processed can be automatically achieved only when the user provides a 2D reference image, meeting the diverse generation needs of 3D model hairstyle areas.

[0077] In one possible implementation, the variation constraint includes a first constraint and a second constraint.

[0078] In step S150, the first hairstyle plane contour point P can be used as a reference. m and the second hairstyle plane outline points r The positional relationship between them and the spatial contour point P′ of the first hairstyle m and the second hairstyle space outline points' r Based on the positional relationship, determine the target spatial contour point P′ after the change (such as TPS change) of the spatial contour point of the first hairstyle. tps Using the first hairstyle spatial contour point P′ m As close as possible to the target spatial contour point P′ tps As the first constraint condition.

[0079] Furthermore, based on the connection relationships between adjacent points in the 3D model to be processed, the second constraint is to maximize the number of points whose connection relationships remain unchanged. Specifically, in the Mesh network structure of the 3D model to be processed, the connection relationships between each point are recorded. During the transformation process in step S160, it is necessary to keep the connection relationships of each point unchanged as much as possible.

[0080] Furthermore, the 3D model to be processed includes a head portion and a hairstyle portion, and the constraint conditions also include a third constraint condition. In step S150, multiple points in the hairstyle portion that are within a preset range from the head portion can be identified as fixed points, and keeping the positions of the fixed points unchanged is used as the third constraint condition. In this way, after adjusting the hairstyle portion, the hairstyle portion and the head portion will not clip through each other.

[0081] In one possible implementation, the hairstyle portion of the 3D model to be processed includes multiple facets. These facets may overlap to form repetitive points, and there may be a lack of topological constraints on their positional relationships. Adjusting the hairstyle portion of the 3D model to be processed may result in the hair facets becoming scattered or the structure becoming distorted.

[0082] Therefore, prior to step S160, the method may also remove overlapping points formed by different facets in the hairstyle portion of the 3D model to be processed, and / or add connection constraint facets between different facets in the hairstyle portion of the 3D model to be processed.

[0083] Specifically, in this embodiment, based on the mesh structure data of the 3D model to be processed, duplicate points formed by the overlap of different facets are removed, and the mapping relationship is stored. After the hairstyle adjustment is completed, the corresponding points are restored according to the mapping relationship. Then, the face structure is added to the deduplicated points to increase the constraints between facets, as shown in the following formula:

[0084] mesh′=Struc(Dedup(mesh),k,perc)

[0085] Wherein, mesh is the original mesh data of the 3D model to be processed, Dedup() is the deduplication operation, Struc() is the operation of adding surface constraint functions, k is the parameter for calculating KNN (K-Nearest Neighbor) points, and perc is the parameter for adding surface structure constraints to a certain proportion of points. If perc is too large, it will lead to too many constraints, and the point distortion operation will have a smaller impact on the structure.

[0086] In one possible implementation, in step S150, points within a set range that are not visible in the preset viewpoint from the 3D model to be processed can be determined as third hairstyle space contour points.

[0087] Then, determine the first positional relationship between the first hair planar contour point and the second hair planar contour point, and determine the second positional relationship between the first hair spatial contour point, the third hair spatial contour point and the second hair spatial contour point.

[0088] Then, based on the first positional relationship and the second positional relationship, determine the target spatial contour point after performing thin-plate spline interpolation transformation on the first hairstyle spatial contour point.

[0089] Thus, during the TPS adjustment of the first hairstyle spatial contour points, the invisible back of the head points of the 3D model to be processed can be affected by the TPS change, thereby ensuring that the constraints on the position of the points in the invisible area are not too large.

[0090] This embodiment also provides an electronic device that can perform the 3D model hairstyle processing method, which may include devices with image processing capabilities such as servers, personal computers, and laptops.

[0091] Please refer to Figure 2 In one possible implementation, the electronic device 100 can communicate with the user terminal 200 via a network, and the electronic device 100 can obtain the reference image from the user terminal 200, and then... Figure 1 The method steps shown are used to adjust the 3D model to be processed.

[0092] In one possible implementation, the electronic device 100 can be a server of a live streaming platform, the user terminal 200 can be a user terminal for viewers or broadcasters, and the 3D model to be processed can be a 3D model used to configure the virtual image of viewers or broadcasters.

[0093] Please refer to Figure 3 , Figure 3 A block diagram of the electronic device 100. The electronic device 100 includes a 3D model hairstyle processing device 110, a machine-readable storage medium 120, and a processor 130.

[0094] The machine-readable storage medium 120 and processor 130 are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The 3D model hairstyle processing device 110 includes at least one software function module that can be stored in the machine-readable storage medium 120 in the form of software or firmware or embedded in the operating system (OS) of the electronic device 100. The processor 130 is used to execute the executable modules stored in the machine-readable storage medium 120, such as the software function modules and computer programs included in the 3D model hairstyle processing device 110.

[0095] The machine-readable storage medium 120 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The machine-readable storage medium 120 is used to store programs, and the processor 130 executes these programs / executable the 3D model hairstyle processing method provided in this embodiment after receiving execution instructions.

[0096] The processor 130 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0097] Please refer to Figure 4 This embodiment also provides a 3D model hairstyle processing device 110, which includes at least one functional module that can be stored in a machine-readable storage medium 120 in software form. Functionally, the 3D model hairstyle processing device 110 may include an image acquisition module 111, a semantic segmentation module 112, a point matching module 113, a point mapping module 114, a constraint determination module 115, and a model adjustment module 116.

[0098] The image acquisition module 111 is used to acquire a reference image and a 3D model to be processed, and to acquire a model planar rendering image of the 3D model to be processed under a preset viewpoint.

[0099] In this embodiment, the image acquisition module 111 can be used to perform... Figure 1 For a detailed description of the image acquisition module 111 shown in step S110, please refer to the description of step S110.

[0100] The semantic segmentation module 112 is used to input the model planar rendering image and the reference image into the image semantic segmentation model respectively to obtain a first hairstyle mask image indicating the hairstyle region in the model planar rendering image and a second hairstyle mask image indicating the hairstyle region in the reference image.

[0101] In this embodiment, the semantic segmentation module 112 can be used to perform... Figure 1 For a detailed description of the semantic segmentation module 112 shown in step S120, please refer to the description of step S120.

[0102] The point matching module 113 is used to determine a first matching point pair on the first hairstyle mask image and the second hairstyle mask image through a dynamic programming algorithm. The first matching point pair includes a first hairstyle planar contour point on the first hairstyle mask image and a second hairstyle planar contour point on the second hairstyle mask image.

[0103] In this embodiment, the point matching module 113 can be used to perform... Figure 1 For a detailed description of the point matching module 113 shown in step S130, please refer to the description of step S130.

[0104] The point mapping module 114 is used to determine the first hairstyle plane contour point and the second hairstyle plane contour point corresponding to the first hairstyle space contour point and the second hairstyle space contour point in the space where the 3D model to be processed is located.

[0105] In this embodiment, the point mapping module 114 can be used to perform... Figure 1 For a detailed description of the point mapping module 114 shown in step S140, please refer to the description of step S140.

[0106] The constraint determination module 115 is used to determine the changing constraint conditions based on the positional relationship between the first hairstyle plane contour point, the second hairstyle plane contour point, the first hairstyle spatial contour point, and the points on the 3D model to be processed.

[0107] In this embodiment, the constraint determination module 115 can be used to perform... Figure 1 For a detailed description of the constraint determination module 115 shown in step S150, please refer to the description of step S150.

[0108] The model adjustment module 116 is used to adjust various points in the hairstyle section of the 3D model to be processed according to the changing constraints.

[0109] In this embodiment, the model adjustment module 116 can be used to perform... Figure 1For a detailed description of the model adjustment module 116 shown in step S160, please refer to the description of step S160.

[0110] In summary, the 3D model hairstyle processing method, apparatus, and electronic device provided in this application determine matching point pairs on the hairstyle mask images of the reference image and the model planar rendering image of the 3D model to be processed, and determine variation constraints based on the relative positional relationship of the matching point pairs to adjust the hairstyle portion of the 3D model to be processed, making the hairstyle area of ​​the 3D model to be processed approach the reference image. Thus, it is possible to automatically adjust the hairstyle area of ​​the 3D model to be processed when only a 2D reference image is provided by the user, meeting the diverse generation needs of 3D model hairstyle areas.

[0111] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0112] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0113] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0115] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing hairstyles in 3D models, characterized in that, The method includes: Acquire a reference image and a 3D model to be processed, and acquire a planar rendered image of the 3D model to be processed from a preset viewpoint; The model planar rendering image and the reference image are respectively input into the image semantic segmentation model to obtain a first hairstyle mask image indicating the hairstyle region in the model planar rendering image and a second hairstyle mask image indicating the hairstyle region in the reference image; A dynamic programming algorithm is used to determine a first matching point pair on the first hairstyle mask image and the second hairstyle mask image. The first matching point pair includes a first hairstyle planar contour point on the first hairstyle mask image and a second hairstyle planar contour point on the second hairstyle mask image. Determine the first hairstyle planar contour point and the second hairstyle planar contour point corresponding to the first hairstyle spatial contour point and the second hairstyle spatial contour point in the space where the 3D model to be processed is located. The variation constraints are determined based on the positional relationships between the first hair planar contour points, the second hair planar contour points, the first hair spatial contour points, and the points on the 3D model to be processed. The points in the hairstyle section of the 3D model to be processed are adjusted according to the changing constraints.

2. The method according to claim 1, characterized in that, Before the step of inputting the model planar rendered image and the reference image into the image semantic segmentation model respectively, the method further includes: Facial landmark alignment adjustment processing is performed on the model planar rendered image and the reference image.

3. The method according to claim 1, characterized in that, The steps of acquiring the reference image and the 3D model to be processed include: Acquire a reference image and identify the hairstyle category of the reference image; The corresponding 3D model to be processed is determined based on the recognition result of the hairstyle category identification.

4. The method according to claim 1, characterized in that, The hairstyle portion of the 3D model to be processed includes multiple facets; prior to the step of determining the variation constraints based on the positional relationship between the first and second hairstyle spatial contour points and the positional relationship between points on the 3D model to be processed, the method further includes: Remove duplicate points formed by the overlapping of different facets in the hairstyle part of the 3D model to be processed; And / or, add connection constraint patches between different patches in the hairstyle section of the 3D model to be processed.

5. The method according to claim 1, characterized in that, The variation constraints include a first constraint and a second constraint; the step of determining the variation constraints based on the positional relationship between the first hairstyle spatial contour points and the second hairstyle spatial contour points and the positional relationship between points on the 3D model to be processed includes: Based on the positional relationship between the first hair planar contour point and the second hair planar contour point and the positional relationship between the first hair spatial contour point and the second hair spatial contour point, the target spatial contour point after performing thin-plate spline interpolation transformation on the first hair spatial contour point is determined, with the first hair spatial contour point being as close as possible to the target spatial contour point as the first constraint condition. The second constraint is to maximize the number of points that maintain the connection relationship unchanged, based on the connection relationship between adjacent points in the 3D model to be processed.

6. The method according to claim 5, characterized in that, The 3D model to be processed includes a head portion and a hairstyle portion; the constraints also include a third constraint; the step of determining the changing constraints based on the positional relationship between the first hairstyle spatial contour points and the second hairstyle spatial contour points and the positional relationship between points on the 3D model to be processed further includes: The third constraint condition is to determine multiple points in the hairstyle section that are within a preset range from the head section as fixed points, so as to keep the position of the fixed points unchanged.

7. The method according to claim 5, characterized in that, The step of determining the target spatial contour point after performing thin-plate spline interpolation transformation on the first hairstyle spatial contour point based on the positional relationship between the first hairstyle planar contour point and the second hairstyle planar contour point and the positional relationship between the first hairstyle spatial contour point and the second hairstyle spatial contour point includes: From the points in the 3D model to be processed that are not visible within the preset viewpoint, determine the points that are within a set range of the first hairstyle space contour points as the third hairstyle space contour points. Determine the first positional relationship between the first hairline planar contour point and the second hairline planar contour point, and determine the second positional relationship between the first hairline spatial contour point, the third hairline spatial contour point, and the second hairline spatial contour point; Based on the first positional relationship and the second positional relationship, determine the target spatial contour point after performing thin-plate spline interpolation transformation on the first hairstyle spatial contour point.

8. A 3D model hairstyle processing device, characterized in that, The 3D model hairstyle processing device includes: The image acquisition module is used to acquire a reference image and a 3D model to be processed, and to acquire a model plane rendering image of the 3D model to be processed under a preset viewpoint. The semantic segmentation module is used to input the model planar rendering image and the reference image into the image semantic segmentation model respectively to obtain a first hairstyle mask image indicating the hairstyle region in the model planar rendering image and a second hairstyle mask image indicating the hairstyle region in the reference image; The point matching module is used to determine a first matching point pair on the first hairstyle mask image and the second hairstyle mask image through a dynamic programming algorithm. The first matching point pair includes a first hairstyle planar contour point on the first hairstyle mask image and a second hairstyle planar contour point on the second hairstyle mask image. The point mapping module is used to determine the first hairstyle plane contour point and the second hairstyle plane contour point corresponding to the first hairstyle space contour point and the second hairstyle space contour point in the space where the 3D model to be processed is located. The constraint determination module is used to determine the changing constraint conditions based on the positional relationship between the first hairstyle plane contour point, the second hairstyle plane contour point, the first hairstyle spatial contour point, and the points on the 3D model to be processed. The model adjustment module is used to adjust each point in the hairstyle part of the 3D model to be processed according to the changing constraints.

9. An electronic device, characterized in that, The method includes a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, which, when executed by the processor, implement the method according to any one of claims 1-7.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions that, when executed by one or more processors, implement the method of any one of claims 1-7.

Images