3D model costume image processing method and device and electronic equipment

Abstract

The application provides a 3D model clothing image processing method and device and electronic equipment. The method comprises: acquiring a to-be-processed image; inputting the to-be-processed image into a clothing category identification model to obtain a clothing category identification result output by the clothing category identification model; determining a corresponding clothing key point detection model according to the clothing category identification result to perform key point detection on the to-be-processed image, and obtaining clothing key points corresponding to each clothing part on the to-be-processed image; and displaying a clothing image in the to-be-processed image to a 3D model surface according to the clothing key points. By selecting a corresponding clothing key point detection model according to an identification result to obtain clothing key points, and then performing segmentation and mapping on the to-be-processed image according to the clothing key points and model mapping key points of the 3D model, a clothing map of the 3D model can be automatically generated according to an input 2D clothing image, the flexibility of 3D model clothing generation is improved, and higher personalized requirements of users can be met.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method, apparatus, and electronic device for processing 3D model clothing images. Background Technology

[0002] With the continuous development of social networking technologies, the application of digital virtual avatars is becoming increasingly widespread, and the demand for diverse and personalized virtual avatars is also increasing. In traditional virtual avatar generation solutions, users can only select patterns from a preset image or clothing texture library to display on the virtual avatar's clothing. This method is difficult to meet the massive and diverse needs for virtual avatar generation. Summary of the Invention

[0003] To overcome the aforementioned shortcomings in the prior art, the purpose of this application is to provide a 3D model clothing image processing method, the method comprising:

[0004] Obtain the image to be processed;

[0005] The image to be processed is input into the clothing category recognition model to obtain the clothing category recognition result output by the clothing category recognition model;

[0006] Based on the clothing category recognition results, a corresponding clothing key point detection model is determined to perform key point detection on the image to be processed, thereby obtaining the clothing key points corresponding to each clothing part on the image to be processed.

[0007] Based on the key points of the clothing, the clothing image in the image to be processed is displayed on the surface of the 3D model.

[0008] In one possible implementation, prior to the step of inputting the image to be processed into the clothing category recognition model, the method further includes:

[0009] The image to be processed is input into the image semantic segmentation model to obtain the clothing region mask image output by the image semantic segmentation model. The clothing region mask image includes a first image region indicating the clothing image and a second image region indicating the non-clothing image.

[0010] Before the step of displaying the clothing image in the image to be processed onto the surface of the 3D model based on the clothing key points, the method further includes:

[0011] Interpolation processing is performed on the key points of the clothing based on the mask image of the clothing area, and / or linear interpolation processing is performed on adjacent key points of the clothing.

[0012] In one possible implementation, prior to the steps of interpolating the clothing key points based on the clothing region mask image and / or performing linear interpolation on adjacent clothing key points, the method further includes:

[0013] Based on the key points of the clothing, the image to be processed is divided into clothing parts, and the outline information of the divided clothing parts is determined.

[0014] The step of interpolating the key points of the clothing based on the clothing region mask image includes:

[0015] Interpolation processing is performed on the key points of the clothing based on the mask image of the clothing area and the outline information of the clothing parts.

[0016] In one possible implementation, prior to the steps of interpolating the clothing key points based on the clothing region mask image and / or performing linear interpolation on adjacent clothing key points, the method further includes:

[0017] The key points of the clothing are adjusted based on the mask image of the clothing area so that the key points of the clothing are located within the first image area.

[0018] In one possible implementation, the step of displaying the clothing image from the image to be processed onto the surface of the 3D model based on the clothing key points includes:

[0019] Obtain a set of UV key points corresponding to the set of 3D surface key points of the 3D model, wherein the set of UV key points includes at least some of the UV key points corresponding to the clothing key points;

[0020] The key points of the clothing are adjusted according to the relative positional relationship of each of the 3D surface key points and the relative positional relationship of each of the UV key points;

[0021] Based on the adjusted key points of the clothing, the clothing image within the first image area is converted into a UV unfolded map;

[0022] Apply the UV unwrapped map to the 3D model.

[0023] In one possible implementation, prior to the step of displaying the clothing image from the image to be processed onto the surface of the 3D model based on the clothing key points, the method further includes:

[0024] The size proportions of the clothing parts are determined based on the key points of the 3D surface of the 3D model;

[0025] The key points of the garment are adjusted according to the size proportions of the garment parts.

[0026] In one possible implementation, the method further includes:

[0027] Color retrieval and / or texture generation are performed on the front image of the clothing in the image to be processed to obtain the back image of the clothing.

[0028] Another objective of this application is to provide a 3D model clothing image processing device, the device comprising:

[0029] The image acquisition module is used to acquire the image to be processed.

[0030] The clothing category recognition module is used to input the image to be processed into the clothing category recognition model and obtain the clothing category recognition result output by the clothing category recognition model;

[0031] The clothing key point recognition module is used to determine the corresponding clothing key point detection model based on the clothing category recognition result, perform key point detection on the image to be processed, and obtain the clothing key points corresponding to each clothing part on the image to be processed.

[0032] The texturing module is used to display the clothing image in the image to be processed onto the surface of the 3D model based on the clothing key points.

[0033] Another objective of this application is to provide an electronic device, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and the machine-executable instructions, when executed by the processor, implement the 3D model clothing image processing method provided in this application.

[0034] Another objective of this application is to provide a machine-readable storage medium, characterized in that the machine-readable storage medium stores machine-executable instructions, which, when executed by one or more processors, implement the 3D model clothing image processing method provided in this application.

[0035] Compared with the prior art, this application has the following beneficial effects:

[0036] This application provides a 3D model clothing image processing method, apparatus, and electronic device. It identifies the clothing category of a user-provided 2D image, then selects a corresponding clothing keypoint detection model based on the identification results to obtain the key points of each part of the clothing. Finally, it segments and maps the image based on these key points and the key points of the 3D model's texture. In this way, it can automatically generate 3D model clothing textures from the input 2D clothing image, improving the flexibility of 3D model clothing generation and meeting users' higher personalization needs. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart illustrating the 3D model clothing image processing method provided in this application embodiment;

[0039] Figure 2 One of the schematic diagrams of key points of clothing provided in the embodiments of this application;

[0040] Figure 3 A second schematic diagram of key clothing points provided in the embodiments of this application;

[0041] Figure 4 A schematic diagram of an electronic device provided in an embodiment of this application;

[0042] Figure 5 This is a schematic diagram of the functional modules of the 3D model clothing image processing device provided in the embodiments of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0044] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0045] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0046] In the description of this application, it should be noted that the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0047] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0048] Please see Figure 1 , Figure 1 This is a flowchart illustrating the steps of a 3D model clothing image processing method provided in this embodiment. The method provided in this embodiment will be described in detail below.

[0049] Step S110: Obtain the image to be processed.

[0050] In this embodiment, the image to be processed can be an image provided by the user containing a target garment that is desired to be displayed on a 3D model. For example, in one example, the image to be processed can be an image that only includes the target garment; in another example, the image to be processed can be a photograph or pattern of a person wearing the target garment.

[0051] Step S120: Input the image to be processed into the clothing category recognition model to obtain the clothing category recognition result output by the clothing category recognition model.

[0052] Since different garments may require significantly different processing methods when applying 3D model textures, in order to improve the accuracy of subsequent image processing, in this embodiment, the image to be processed can be input into the pre-trained garment category recognition model. The model first determines the type of the target garment in the image, and then further refines the processing based on the type of the target garment. The output of the garment category recognition model can include garment category recognition results such as short-sleeved shirts, shorts, trousers, skirts, and sundresses.

[0053] Step S130: Based on the clothing category recognition result, determine the corresponding clothing key point detection model to perform key point detection on the image to be processed, and obtain the clothing key points corresponding to each clothing part on the image to be processed.

[0054] In this embodiment, multiple clothing keypoint detection models can be pre-trained for keypoint detection of different types of clothing. In step S130, based on the clothing category indicated by the clothing category recognition result, the corresponding clothing keypoint detection model can be determined to perform keypoint detection on the image to be processed. In this way, the corresponding clothing keypoints can be more accurately determined for different categories of clothing, thereby improving the accuracy of clothing segmentation and mapping. Furthermore, by using different clothing keypoint detection models to perform keypoint detection on different types of clothing, it is possible to more flexibly expand the scope of inspection when adding unconventional clothing types later. For example, when it is necessary to add unconventional and unusual clothing, armor, etc., from scenes such as anime and games, it is not necessary to adjust the already trained clothing keypoint recognition models; only the clothing keypoint detection model corresponding to the new clothing type needs to be added.

[0055] The key points of the clothing can be pre-set key points of the outer contour of the clothing and key points for dividing the clothing parts. The key points of the outer contour of the clothing can include key points that indicate the outer contour of the clothing, such as the neckline, the hem of the sleeve, and the bottom of the garment. The key points for dividing the clothing parts can include the junction of the sleeve and the body of the garment, the junction of the skirt body and the hem of the skirt, etc.

[0056] Step S140: Display the clothing image in the image to be processed onto the surface of the 3D model according to the clothing key points.

[0057] In this embodiment, after determining the key points of the clothing, the positional correspondence between each position of the clothing image in the image to be processed and the surface of the 3D model can be determined based on the key points of the clothing, thereby displaying the clothing image in the image to be processed onto the surface of the 3D model.

[0058] For example, after determining the key points of the clothing, the correspondence between each key point of the clothing and the UV unwrapping point set of the 3D model can be determined. Then, according to the correspondence, the clothing image in the image to be processed is subjected to the corresponding warp operation to obtain a UV unwrapping map that matches the 3D model. Then, according to the correspondence between the UV unwrapping point set and the 3D surface key points of the 3D model, the UV unwrapping map is displayed on the surface of the 3D model.

[0059] Optionally, the surface contours of the corresponding 3D models may differ for different types of clothing. Therefore, in this embodiment, the corresponding 3D model can be determined based on the clothing type identification result in step S120, such as the 3D model of trousers, the 3D model of a dress, the 3D model of a trench coat, etc. Then, in step S140, the relationship between the changes of the clothing key points and the corresponding model texture key points on the 3D model surface is determined.

[0060] Based on the above design, in this embodiment, the 2D image to be processed provided by the user is used to identify the clothing category. Then, according to the identification results, a corresponding clothing key point detection model is selected to obtain the clothing key points of each part of the clothing. Then, the image to be processed is segmented and textured according to the clothing key points and the model texture key points of the 3D model. In this way, the clothing texture of the 3D model can be automatically generated from the input 2D clothing image, which improves the flexibility of 3D model clothing generation and can meet the user's higher personalization needs.

[0061] In some scenarios, the image to be processed input by the user can be an image of a person wearing the target clothing (e.g., a photograph of a person). Therefore, in some possible implementations, after obtaining the image to be processed in step S110, the image to be processed can be first input into an image semantic segmentation model to obtain the clothing region mask image output by the image semantic segmentation model. The clothing region mask image includes a first image region indicating the clothing image and a second image region indicating the non-clothing image. Furthermore, while determining the clothing region mask image, the position of the clothing image in the image to be processed can also be determined as a Region of Interest (ROI).

[0062] Specifically, the semantic segmentation result output by the image semantic segmentation model may have edge jaggedness due to recognition accuracy issues, or the semantic segmentation result output by the image semantic segmentation model may contain free blocks when there are multiple clothing images in the image to be processed. In this embodiment, post-processing algorithms such as edge smoothing and free block removal can be performed on the semantic segmentation result output by the image semantic segmentation model, and then the ROI recognition operation can be performed to bring better key point detection accuracy. Finally, a single semantic image mask is obtained for each target clothing image as the clothing region mask image img. mask Specifically, as shown in formula (1):

[0063] img mask =Pre(E seg (img rgb )) (1)

[0064] box(x1,x2,y1,y2)=F roi (img mask (2)

[0065] Among them, img rgb For the image to be processed, E seg The image semantic segmentation model is defined as follows; Pre is the post-processing function for the semantic segmentation output, used to smooth edges and remove free blocks; img maskThe mask image for the clothing area. F roi The function to obtain the ROI, where x1, x2, y1, y2 are the mask images of the clothing region (img). mask The coordinate parameters of the four vertices of the box representing the region of interest.

[0066] For the image to be processed rgb Similarly, perform the ROI extraction operation and compare it with the clothing area mask image (img). mask Composite image (img) of transparent PNGs with alpha channels, created by stitching together elements. rgba Specifically, as shown in formula (3):

[0067] img rgba =concat(crop(img) rgb ,box),crop(img mask ,box)) (3)

[0068] Here, `crop` is the operation to extract the Region of Interest (ROI), and `concat` is the operation to stitch together a transparent PNG image with an alpha channel.

[0069] Thus, by using the image semantic segmentation model to determine the clothing region mask image indicating the area where the clothing image is located, and to determine the ROI region of the clothing image on the image to be processed, it is possible to facilitate further image processing, reduce interference from irrelevant images, reduce subsequent computational load, and improve the accuracy of subsequent image processing.

[0070] In some possible implementations, in step S120, a universal classification network can be used as the clothing category recognition model for the image in the ROI region. roi Perform classification and recognition to obtain the key(img) of the clothing. roi Specifically, it can be obtained through formula (4):

[0071]

[0072] Wherein, Cls is the clothing type recognition model, used to output the image (img) in the ROI region. roi The clothing category recognition result; Key() is the clothing key point detection model, and its subscript is the clothing category recognition result of the clothing type recognition model. It is used to access the corresponding clothing key point detection model for the image img in the ROI region according to the clothing category recognition result. roi Perform and output the key(img) of the clothing. roi ).

[0073] In some possible implementations, when the clothing region mask image is obtained through the image semantic segmentation model, the clothing key points can be adjusted according to the clothing region mask image before step S140 so that the clothing key points are located within the first image region.

[0074] Specifically, in this embodiment, theoretically, all clothing key points detected by the clothing key point detection model should be located within the first image region of the clothing region mask image (i.e., within the region where the clothing image is located). However, in practice, due to issues such as image interference, limitations in the training samples of the clothing key point detection model, or limitations in the recognition ability of the clothing key point detection model itself, some clothing key points may not be located within the first image region. Therefore, in this embodiment, the clothing key points can be adjusted according to the clothing region mask image to ensure that the clothing key points are located within the first image region, thereby guaranteeing accurate segmentation of the clothing texture in the subsequent process.

[0075] Specifically, when adjusting the key points of the clothing, some of these key points possess specific semantic information (e.g., representing sleeve corners, garment hems, etc.). These key points are typically points on the boundary of the first image region in the clothing region mask image obtained by the image semantic segmentation model. Furthermore, some key points are often corner points with significant slope changes. Considering the symmetry of the clothing and some prior information, these key points can be combined to fine-tune the final key points of the clothing. For non-inflection point key points, the adjustment method can be as shown in equation (5). For obvious corner point key points, the adjustment method can be as shown in equation (6).

[0076] key′=FNP(FCT(img rgba ),Refine(key)) (5)

[0077] key′=FNP(FC(FCT(img rgba )),Refine(key)) (6)

[0078] FCT() is used to obtain the synthetic image img. rgba The functions for dividing the boundary are: FC(), FNP(), and Refine(). FC() is used to obtain the set of important inflection points in the boundary. FNP() is used to find the nearest point. Refine() is used to adjust the key points based on prior information (e.g., the upper boundary of the contour near the upper neckline, the lower boundary of the contour near the lower trouser hem, the fact that the left and right cuffs are usually the same height).

[0079] Thus, by combining the mask image of the clothing area...

[0080] In some possible implementations, the number of clothing key points determined by the clothing key point detection model may be small. This could lead to issues such as cutouts, distortions, or misalignment at the seams of the clothing in the 3D model when relying solely on these key points for texturing. Therefore, after obtaining the clothing region mask image through the image semantic segmentation model, before step S140, interpolation processing can be performed on the clothing key points based on the clothing region mask image, and / or linear interpolation processing can be performed on adjacent clothing key points. This increases the number of clothing key points, which is beneficial for subsequent texturing operations.

[0081] For example, please refer to Figure 2 , Figure 2 The clothing key points determined by the clothing key point detection model may include some key points that are not located on the boundary of the first image region, and the current number of clothing key points may be relatively small.

[0082] Based on this, the key points of the clothing can be adjusted by combining the boundary of the first image region in the clothing region mask image.

[0083] Then, please refer to Figure 3 Interpolation can be performed along the boundary indicated by the first image region in the clothing area mask image; linear interpolation can be performed for two adjacent but far apart clothing key points that are not on the boundary; or adjustments or further interpolation can be performed after linear interpolation based on the clothing key points located on the boundary.

[0084] Furthermore, the image to be processed can first be segmented into clothing parts based on the clothing key points to determine the outline information of the segmented clothing parts. Then, the clothing key points can be interpolated based on the clothing region mask image and the clothing part outline information. That is, in the interpolation process, in addition to considering the overall outer outline of the clothing, the outline information of each part of the clothing can also be added, so that the information that the clothing key points can express after interpolation is richer.

[0085] Specifically, in some virtual idol anime texture images, the image is usually composed of multiple blocks. In this embodiment, the clothing can be segmented by combining the key points of the clothing with fixed semantics, the contour information indicated by the clothing area mask image, and the fitted curve. For example, the left and right sleeves and body areas can be segmented, as can the left and right front and back trouser legs, the skirt top and bottom, and the sleeves. In this way, some of the clothing key points (such as the cuff dividing point of a long sleeve) can be further interpolated based on the contour information of the clothing parts obtained after segmentation.

[0086] Since the display effect of a 2D clothing image on a 3D model is related to the density and position information of the 3D model's wiring (for example, an image that is normal on a 2D texture map may be stretched or shrunk to an extreme extent on a 3D model due to wiring), in some possible implementations of this embodiment, in step S140, a set of UV key points corresponding to the set of 3D surface key points of the 3D model can be obtained first. The set of UV key points includes at least some UV key points corresponding to the clothing key points. Then, the clothing key points are adjusted according to the relative positional relationships of the 3D surface key points and the relative positional relationships of the UV key points. Then, the clothing image in the first image area is converted into a UV unwrapped map according to the adjusted clothing key points, and the UV unwrapped map is applied to the 3D model.

[0087] Specifically, in this embodiment, after obtaining the set of UV key points corresponding to the set of 3D surface key points of the 3D model, the length of the first line segment between each adjacent 3D surface key point and the length of the second line segment between each adjacent UV key point can be calculated.

[0088] Then, using the line segment with the smallest deformation as the standard, the stretching ratio between each first line segment and its corresponding second line segment is determined. Based on the stretching ratio corresponding to each UV keypoint, the movement direction corresponding to each UV keypoint is determined, and the set of UV keypoints is moved and adjusted accordingly to obtain the adjusted set of UV keypoints. Specifically, using the line segment with the smallest deformation as the standard, the stretching ratio corresponding to the second line segment connected to the UV keypoint can characterize the ratio of compression or stretching required between the UV keypoint and other UV keypoints when mapping the UV keypoint to the surface of the 3D model. Based on the relative positional relationship between the UV keypoint and each adjacent UV keypoint and the stretching ratio, multiple stretching movement vectors can be determined, and the movement vector corresponding to the UV keypoint can be determined based on the sum of the multiple stretching vectors. Then, based on this movement vector, with the goal of reducing the movement vector of each UV keypoint, multiple iterative movement adjustments are performed with small displacements to obtain the adjusted UV keypoints.

[0089] Next, based on the correspondence between the adjusted UV key points and the clothing key points, the clothing image within the first image area is warped to obtain a UV unfolded map. Then, based on the positional changes of each UV key point before and after adjustment, the UV unfolded map is adjusted to match the original positions of the UV key points. Finally, the UV unfolded map is applied to the 3D model.

[0090] In some possible implementations, the target clothing pattern given in the image to be processed may differ from the overall pattern of the 3D model. Therefore, in this embodiment, in step S130, the size ratio of the clothing part can be determined according to the model texture key points of the 3D model, and then the clothing key points can be adjusted according to the size ratio of the clothing part.

[0091] For example, the corresponding sleeve length, garment length, aspect ratio, and neckline length ratio can be determined based on the key points of the 3D model's texture map. Then, the correspondence between the garment's key points and the 3D model's texture map key points can be determined based on these ratios. In step S140, the garment texture can be warped and adjusted proportionally to make the garment texture match the garment pattern of the 3D model.

[0092] In some possible implementations, some of the images to be processed may only include the pattern or texture of the front of the clothing, while the 3D model requires clothing pattern or texture images from multiple perspectives. Therefore, in this embodiment, color retrieval and / or texture generation are performed based on the front image of the clothing in the images to be processed to obtain the back image of the clothing. Thus, in subsequent...

[0093] Please refer to Figure 4 This embodiment also provides an electronic device 100, which can be a server, personal computer, laptop computer, or other device with image processing capabilities. The electronic device 100 includes a 3D model clothing image processing device 110, a machine-readable storage medium 120, and a processor 130.

[0094] The machine-readable storage medium 120, processor 130, and communication unit 140 are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The 3D model clothing image processing device 110 includes at least one software function module that can be stored in the machine-readable storage medium 120 in the form of software or firmware or embedded in the operating system (OS) of the electronic device 100. The processor 130 is used to execute the executable modules stored in the machine-readable storage medium 120, such as the software function modules and computer programs included in the 3D model clothing image processing device 110.

[0095] The machine-readable storage medium 120 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The machine-readable storage medium 120 is used to store programs, and after receiving execution instructions, the processor 130 executes the 3D model clothing image processing method provided in this embodiment.

[0096] The processor 130 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0097] Please refer to Figure 5 This embodiment also provides a 3D model clothing image processing device 110, which includes at least one functional module that can be stored in a machine-readable storage medium 120 in software form. Functionally, the 3D model clothing image processing device 110 may include an image acquisition module 111, a clothing category recognition module 112, a clothing key point recognition module 113, and a texture mapping module 114.

[0098] The image acquisition module 111 is used to acquire the image to be processed.

[0099] In this embodiment, the image acquisition module 111 can be used to perform... Figure 1 For a detailed description of the image acquisition module 111 shown in step S110, please refer to the description of step S110.

[0100] The clothing category recognition module 112 is used to input the image to be processed into the clothing category recognition model and obtain the clothing category recognition result output by the clothing category recognition model.

[0101] In this embodiment, the clothing category identification module 112 can be used to perform... Figure 1 For a detailed description of the clothing category identification module 112 shown in step S120, please refer to the description of step S120.

[0102] The clothing key point recognition module 113 is used to determine the corresponding clothing key point detection model based on the clothing category recognition result, perform key point detection on the image to be processed, and obtain the clothing key points corresponding to each clothing part on the image to be processed.

[0103] In this embodiment, the clothing key point recognition module 113 can be used to perform... Figure 1 For a detailed description of the clothing key point recognition module 113 shown in step S130, please refer to the description of step S130.

[0104] The texture module 114 is used to obtain clothing textures from the image to be processed based on the clothing key points, and to display the clothing textures on the 3D model according to the change relationship.

[0105] In this embodiment, the mapping module 114 can be used to perform... Figure 1 For a detailed description of the mapping module 114 shown in step S140, please refer to the description of step S140.

[0106] In summary, the 3D model clothing image processing method, apparatus, and electronic device provided in this application identify the clothing category of the 2D image to be processed provided by the user, then select an appropriate clothing key point detection model to obtain the clothing key points of each part of the clothing based on the identification results, and then segment and texture the image to be processed based on the clothing key points and the model texture key points of the 3D model. In this way, clothing textures for 3D models can be automatically generated from the input 2D clothing image, improving the flexibility of 3D model clothing generation and meeting users' higher personalization needs.

[0107] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0108] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0109] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0111] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for processing 3D model clothing images, characterized in that, The method includes: Multiple clothing keypoint detection models were pre-trained for different clothing categories. Obtain the image to be processed; The image to be processed is input into the clothing category recognition model to obtain the clothing category recognition result output by the clothing category recognition model; Based on the clothing category recognition result, the corresponding clothing key point detection model is determined to perform key point detection on the image to be processed, and obtain the clothing key points corresponding to each clothing part on the image to be processed. Determine the corresponding 3D model based on the clothing category recognition results; Obtain a set of UV key points corresponding to the set of 3D surface key points of the 3D model, wherein the set of UV key points includes at least some of the UV key points corresponding to the clothing key points; Based on the relative positional relationships of the 3D surface key points and the UV key points, the clothing key points are adjusted; the clothing image is then converted into a UV unfolded image based on the adjusted clothing key points. Apply the UV unwrapped map to the surface of the 3D model.

2. The method according to claim 1, characterized in that, Before the step of inputting the image to be processed into the clothing category recognition model, the method further includes: The image to be processed is input into the image semantic segmentation model to obtain the clothing region mask image output by the image semantic segmentation model. The clothing region mask image includes a first image region indicating the clothing image and a second image region indicating the non-clothing image. Before the step of obtaining the set of UV key points corresponding to the set of 3D surface key points of the 3D model, the method further includes: Interpolation processing is performed on the key points of the clothing based on the mask image of the clothing area, and / or linear interpolation processing is performed on adjacent key points of the clothing.

3. The method according to claim 2, characterized in that, Before the steps of interpolating the clothing key points based on the clothing region mask image and / or performing linear interpolation on adjacent clothing key points, the method further includes: Based on the key points of the clothing, the image to be processed is divided into clothing parts, and the outline information of the divided clothing parts is determined. The step of interpolating the key points of the clothing based on the clothing region mask image includes: Interpolation processing is performed on the key points of the clothing based on the mask image of the clothing area and the outline information of the clothing parts.

4. The method according to claim 2, characterized in that, Before the steps of interpolating the clothing key points based on the clothing region mask image and / or performing linear interpolation on adjacent clothing key points, the method further includes: The key points of the clothing are adjusted based on the mask image of the clothing area so that the key points of the clothing are located within the first image area.

5. The method according to claim 1, characterized in that, Before the step of obtaining the set of UV key points corresponding to the set of 3D surface key points of the 3D model, the method further includes: The size proportions of the clothing parts are determined based on the key points of the 3D surface of the 3D model; The key points of the garment are adjusted according to the size proportions of the garment parts.

6. The method according to claim 1, characterized in that, The method further includes: Color retrieval and / or texture generation are performed on the front image of the clothing in the image to be processed to obtain the back image of the clothing.

7. A 3D model clothing image processing device, characterized in that, The device includes: The image acquisition module is used to acquire the image to be processed. The clothing category recognition module is used to input the image to be processed into the clothing category recognition model and obtain the clothing category recognition result output by the clothing category recognition model; The clothing key point recognition module is used to pre-train multiple clothing key point detection models for key point detection of different clothing categories; it is also used to determine the corresponding clothing key point detection model according to the clothing category recognition result to perform key point detection on the image to be processed, and obtain the clothing key points corresponding to each clothing part on the image to be processed. The texturing module is used to obtain a set of UV key points corresponding to the set of 3D surface key points of the 3D model, wherein the set of UV key points includes at least some UV key points corresponding to the clothing key points; adjust the clothing key points according to the relative positional relationship of each of the 3D surface key points and the relative positional relationship of each of the UV key points; convert the clothing image into a UV unwrapped map according to the adjusted clothing key points; and apply the UV unwrapped map to the surface of the 3D model.

8. An electronic device, characterized in that, The method includes a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, which, when executed by the processor, implement the method according to any one of claims 1-6.

9. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions that, when executed by one or more processors, implement the method according to any one of claims 1-6.

Images