Method for generating three-dimensional clothing chartlet and related device

Abstract

The invention relates to a method for generating a three-dimensional clothing map and a related device. The method for generating the three-dimensional clothing chartlet comprises the steps that first information from a user is acquired, and the first information indicates features of a target three-dimensional clothing chartlet expected by the user; determining a data type of the first information, wherein the data type of the first information comprises a text and an image; according to the data type of the first information, inputting the first information into a first image generation model to generate a clothing front view; inputting the garment front view into a second image generation model to generate a garment back view; inputting the garment front view and the garment back view into a fusion model to generate a two-dimensional garment image; and inputting the two-dimensional clothing image into the back projection model to generate a target three-dimensional clothing map.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a method for generating three-dimensional clothing maps, an electronic device, a non-transitory storage medium, and a computer program product. Background Art

[0002] In today's digital world, the development of virtual reality (VR) technology enables users to experience immersive virtual environments by installing specific applications (APPs). These virtual environments can not only provide realistic scene rendering, but also contain rich virtual characters, such as simulated people, cartoon characters or other fictional characters. Summary of the invention

[0003] A brief overview of the disclosure is given below in order to provide a basic understanding of some aspects of the disclosure. However, it should be understood that this overview is not an exhaustive overview of the disclosure. It is not intended to identify the key or important parts of the disclosure, nor is it intended to limit the scope of the disclosure. Its purpose is simply to give some concepts of the disclosure in a simplified form as a prelude to a more detailed description given later.

[0004] According to a first aspect of the present disclosure, a method for generating a three-dimensional clothing map is provided, comprising: obtaining first information from a user, the first information indicating features of a target three-dimensional clothing map desired by the user; determining a data type of the first information, the data type of the first information comprising text and an image; according to the data type of the first information, inputting the first information into a first image generation model to generate a clothing front view; inputting the clothing front view into a second image generation model to generate a clothing back view; inputting the clothing front view and the clothing back view into a fusion model to generate a two-dimensional clothing image; and inputting the two-dimensional clothing image into a back projection model to generate a target three-dimensional clothing map.

[0005] According to a second aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory storing computer executable instructions, which, when executed by the processor, enables the processor to execute the method for generating a three-dimensional clothing map according to the first aspect of the present disclosure.

[0006] According to a third aspect of the present disclosure, a non-volatile storage medium having computer executable instructions stored thereon is provided. When the computer executable instructions are executed by a processor, the processor executes the method for generating a three-dimensional clothing map according to the first aspect of the present disclosure.

[0007] According to a fourth aspect of the present disclosure, a computer program product is provided. The computer program product includes instructions. When the instructions are executed by a processor, the method for generating a three-dimensional clothing map according to the first aspect of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The foregoing and other features and advantages of the present disclosure will become apparent from the following description of the embodiments of the present disclosure in conjunction with the accompanying drawings. The accompanying drawings are incorporated herein and form a part of the specification, and are further used to explain the principles of the present disclosure and enable those skilled in the art to make and use the present disclosure. Among them:

[0009] Figure 1 A flowchart of a method for generating a three-dimensional clothing map according to some embodiments of the present disclosure is shown;

[0010] Figure 2A and Figure 2B A flowchart illustrating a non-limiting example process of a method for generating a three-dimensional garment map according to some embodiments of the present disclosure;

[0011] Figure 3 A schematic block diagram of an electronic device according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A schematic block diagram of a computer system is shown on which embodiments of the present disclosure may be implemented.

[0013] Note that in the embodiments described below, sometimes the same reference numerals are used in common between different drawings to represent the same parts or parts with the same functions, and their repeated descriptions are omitted. In some cases, similar numbers and letters are used to represent similar items, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0014] For ease of understanding, the position, size, range, etc. of each structure shown in the drawings and the like may not represent the actual position, size, range, etc. Therefore, the present disclosure is not limited to the position, size, range, etc. disclosed in the drawings and the like. DETAILED DESCRIPTION

[0015] Various exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0016] The following description of at least one exemplary embodiment is in fact merely illustrative and is in no way intended to limit the present disclosure and its application or use. That is, the structures and methods herein are shown in an exemplary manner to illustrate different embodiments of the structures and methods in the present disclosure. However, those skilled in the art will appreciate that they merely illustrate exemplary ways of the present disclosure that can be implemented, rather than exhaustive ways. In addition, the drawings need not be drawn to scale, and some features may be enlarged to illustrate the details of specific components.

[0017] In addition, technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0018] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0019] In VR technology, the interaction between users and virtual characters is a key part of the augmented reality experience. In many current applications, users can personalize virtual characters so that the appearance and style of each character can reflect the user's personal preferences.

[0020] However, clothing selection in most current apps is limited to a pre-configured clothing library within the app. Although this approach can provide a certain degree of personalization, it is often difficult to meet users' needs for diversity and innovation due to the size and content of the clothing library.

[0021] To this end, the present disclosure provides a method that can generate a three-dimensional clothing map that meets the user's expectations based on the information input by the user, thereby allowing the user to not only select the three-dimensional clothing map from limited preset options, but also generate and adjust the three-dimensional clothing map according to the user's specific needs and preferences, thereby greatly expanding the dressing flexibility of the virtual character and providing the user with a richer and more personalized virtual dressing experience.

[0022] The method for generating a three-dimensional clothing map according to various embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings. It is understood that the actual method for generating a three-dimensional clothing map may also include other steps, but in order to avoid blurring the main points of the present disclosure, these other steps are not discussed herein and are not shown in the accompanying drawings.

[0023] Figure 1 FIG. 1 is a flowchart of a method 100 for generating a three-dimensional clothing map according to some embodiments of the present disclosure. Figure 1 As shown, the method 100 includes steps S102 to S112.

[0024] In step S102, first information is obtained from a user, where the first information indicates features of a target three-dimensional clothing map desired by the user.

[0025] In step S104, the data type of the first information is determined, and the data type of the first information includes text and image.

[0026] At step S106, the first information is input into a first image generation model to generate a clothing front view according to the data type of the first information.

[0027] At step S108, the garment front view is input into the second image generation model to generate a garment back view.

[0028] At step S110, the clothing front view and the clothing back view are input into the fusion model to generate a two-dimensional clothing image;

[0029] At step S112, the two-dimensional clothing image is input into the back-projection model to generate a target three-dimensional clothing map.

[0030] In some examples, the user can upload a photo as a blueprint of the target three-dimensional clothing map or upload a text to describe the various features of the target three-dimensional clothing map. Of course, the user can also input a voice or a picture containing only text to convert it into a corresponding text according to the voice or picture. In a non-limiting embodiment, assuming that the user expects to generate a three-dimensional clothing of "Hanfu" as a virtual image, the user can upload a favorite Hanfu image or directly input (text input, voice input or picture input) a text describing the characteristics of the Hanfu, wherein the Hanfu image can include a design drawing of the Hanfu and / or a multi-angle photo of the real object, etc., and the text can include the style, size, material, color, pattern, etc. related to the Hanfu. It is understandable that the user can also input a combination of image and text.

[0031] In some embodiments, inputting the first information into the first image generation model to generate a clothing front view may include: extracting the art style features desired by the user based on the first information, and inputting the art style features and the features of the target three-dimensional clothing map into the first image generation model. In this embodiment, a style plug-in model for modulating the art style of the image is incorporated into the first image generation model as a control weight.

[0032] When generating a clothing front view, since the products that actually use these three-dimensional clothing usually have one or more specific art styles (for example, for a product in the field of animation, the art style of the product may include a cartoon style), the control weight matrix of the style insertion model can be combined with the original weight matrix of the first image generation model to readjust the weight matrix of the first image generation model so that the generated clothing front view can conform to the art style of the product and avoid conflicts with the art style of the product, thereby improving the user experience of the product.

[0033] In addition, the clothing style indicated by the first information input by the user is uncontrollable, or in other words, sometimes the art style expected by the user may be an art style that is not supported by the product. In this case, even if the unsupported art style features are input into the first image generation model, the style insertion model can readjust the weight matrix of the first image generation model to ensure that the art style of the generated clothing front view can still meet the art style of the product, thereby ensuring the user's experience of using the product.

[0034] Specifically, in some embodiments, the style insertion model can be trained using the first training data, wherein the first training data includes a plurality of sample clothing images having a preset art style. In this embodiment, the first training data trains the style insertion model in a low rank adaptation (Lora) manner. For example, 10 clothing images pre-made by a designer can be rendered into 10 corresponding clothing front views, and these clothing front views can be used as positive samples for defining the art style, and these positive samples can be used to train the style insertion model in a Lora manner. Thus, in the reasoning process of generating the clothing front view, the weights of the style insertion model are integrated into the first image generation model to control the art style of the generated clothing front view to be consistent with the art style of the product.

[0035] In some embodiments, when the data type of the first information is text, the first image generation model includes a diffusion model and a ControlNet model. Additionally or alternatively, since the style insertion model is integrated into the first image generation model as a control weight, it can be considered that the first image generation model also includes a style insertion model. At this time, from a functional perspective, the first image generation model can be called a text image model.

[0036] Specifically, inputting the first information into the first image generation model to generate a clothing front view may include: inputting the first information into a diffusion model, and adjusting the parameters of the diffusion model based on the ControlNet model and the style plug-in model to generate a clothing front view. Here, the ControlNet model is a deep learning model that aims to enhance the capabilities of the diffusion model by providing additional control information. The ControlNet model can make the generated image more accurate and more in line with user requirements by processing and fusing different types of control inputs, wherein the input type and / or content includes posture, key points, Canny images (edge ​​images), etc., to improve the diffusion model to generate a clothing front view that meets the requirements.

[0037] In some examples, when the first information input by the user is text, the Canny image of the clothing template obtained by the traditional computer vision (CV) algorithm can be used as guidance information of the ControlNet model to generate a front view of the clothing that conforms to the text description in the first information, wherein the CV algorithm may include, for example, Scale-Invariant Feature Transform (SIFT), the machine vision tool YOLO (You Only Look Once), the Canny algorithm, and the like.

[0038] In some other embodiments, when the data type of the first information is an image, the first image generation model may include an image feature extraction model, a diffusion model, and a ControlNet model. Of course, since the style insertion model is integrated into the first image generation model as a control weight, it can be considered that the first image generation model also includes a style insertion model. At this time, the first image generation model can be called an image generation model from a functional perspective. Additionally, in some embodiments, the image feature extraction model may include a diffusion inversion model.

[0039] Further, it can be understood that when the first information input by the user includes both an image and text, the first image generation model may include an image-generated image model and a text-generated image model.

[0040] In some embodiments, inputting the first information into a first image generation model to generate a front view of a garment includes: inputting the first information into an image feature extraction model to generate an image prompt.

[0041] In some embodiments, after the image prompt word is generated, the features contained in the first information input by the user have been converted from image type to text type. At this time, the required image can be generated by referring to the processing method of the aforementioned text image model; that is, inputting the first information into the first image generation model to generate a clothing front view also includes: inputting the image prompt word into the diffusion model, and adjusting the parameters of the diffusion model based on the ControlNet model and the style plug-in model to generate a clothing front view. In some non-limiting examples, if the first information input by the user includes a clothing image, the clothing image can be converted into an image prompt word using a diffusion inversion model, such as a user uploading an image of a physical clothing object, thereby generating a feature description of the clothing in the image, such as "black, cotton, sportswear", etc. Further, the image prompt word is used as the input of the diffusion model, and the ControlNet model is used as a guide to generate a clothing front view that conforms to the target three-dimensional image.

[0042] On the other hand, in some embodiments, inputting the first information into the first image generation model to generate a front view of the clothing also includes: performing pose estimation on the image in the first information to determine the deviation between the pose of the image in the first information and the pose of the front view; adjusting the redrawing weight of the ControlNet model based on the deviation between the pose of the image in the first information and the pose of the front view; inputting the image prompt words into the diffusion model and adjusting the parameters of the diffusion model based on the ControlNet model and the style plug-in model to generate the front view of the clothing.

[0043] In some non-limiting examples, when a user inputs a clothing image, since the user input is uncontrollable, the clothing image input by the user often includes a clothing side view in addition to the clothing front view. Therefore, the input clothing image can be subjected to posture estimation via a posture estimation algorithm to estimate whether the input clothing image is facing the camera head-on, wherein the posture estimation algorithm includes methods such as triangulation and feature matching. If the input clothing image is not a front view or is not facing the camera head-on, it is necessary to adjust the redrawing ratio of the first image generation model according to the deviation to enhance the effect of the clothing front view generated by the first image generation model. Thus, even if the clothing image input by the user is not a clothing front view, a better clothing front view can be generated based on the clothing image input by the user.

[0044] Specifically, in some embodiments, in response to the deviation between the pose of the image in the first information and the front view pose being less than 30°, the redrawing weight of the ControlNet model is adjusted to 0.3; in response to the deviation between the pose of the image in the first information and the front view pose being not less than 30° and not greater than 60°, the redrawing weight of the ControlNet model is adjusted to 0.5; in response to the deviation between the pose of the image in the first information and the front view pose being greater than 60°, the redrawing weight of the ControlNet model is adjusted to 0.8. It can be understood that the classification of the deviation between the pose of the image in the first information and the front view pose can have finer or coarser classifications, and the redrawing weight corresponding to each classification can be reset according to actual needs.

[0045] After obtaining the front view of the clothing through the first information input by the user and the first image generation model, the front view of the clothing can be input into the second image generation model to generate the back view of the clothing.

[0046] In some embodiments, the second image generation model may be trained using second training data, the second training data including a plurality of front views of clothing as samples and a corresponding plurality of back views of clothing as labels of the samples.

[0047] In some examples, a large number of three-dimensional clothing maps can be selected and the clothing front view and clothing back view of these three-dimensional clothing maps can be rendered respectively. When training the second image generation model, the clothing front view is used as input, and the clothing back view is used as the model prediction target, and training is performed in a supervised paradigm. For example, the training goal is to restore the back view of the clothing. During the training process, an image with a pixel size of 1024×512 is input, where the 512×512 part on the left is the clothing front view, and the 512×512 part on the right is the area to be generated. The area to be generated is filled with pure black pixels, and the pixel-level mean square error loss function (L2 Loss) is used for supervision.

[0048] Therefore, using the clothing front view as a condition to generate the corresponding clothing back view can better improve the problem of inconsistent colors and textures on the front and back of the clothing.

[0049] After obtaining the clothing front view and the clothing back view, the clothing front view and the clothing back view may be input into a fusion model to generate a two-dimensional clothing image.

[0050] In some embodiments, the fusion model includes a generative adversarial network (GAN) for image restoration. In this embodiment, inputting the front view and the back view of the clothing into the fusion model to generate a two-dimensional clothing image includes: inputting the front view and the back view of the clothing into the fusion model to generate an initial two-dimensional clothing image; detecting defects in the initial two-dimensional clothing image; in response to detecting that the initial two-dimensional clothing image has defects, performing image restoration on the initial two-dimensional clothing image based on the generative adversarial network to obtain a two-dimensional clothing image.

[0051] In the process of generating two-dimensional clothing images, there will be unnatural seams at locations such as the sides and cuffs of the two-dimensional clothing images. In addition, due to the uncontrollability of user input, the input image may be blocked, resulting in defects such as patches and noise in the generated two-dimensional clothing images. Therefore, the fusion model can automatically repair the defects in the initial two-dimensional clothing image by using the characteristics of clothing symmetry and regularity. For example, assuming that there are some noise points on the initial two-dimensional clothing image, the fusion model can first locate the pixel area near the noise point (for example, 9 pixels centered on the noise point) and then find the pixel area near the noise point. 9 window), then calculate the pixel average of the pixel area and determine the difference between the value of each pixel in the pixel area and the pixel average. When the difference between the value of each pixel in the pixel area and the pixel average does not exceed the preset threshold (i.e., the color difference is small), the pixel value of the noise point can be updated with the pixel average. When the difference between the value of each pixel in the pixel area and the pixel average exceeds the preset threshold (i.e., the color difference is large), the fusion model can use the clothing symmetry principle to sample in the symmetrical area of ​​the clothes (such as the left / right hand sleeves) to update the pixel value of the noise point.

[0052] In some embodiments, the fusion model can be trained using third training data, which includes multiple two-dimensional clothing images with defects as samples and multiple two-dimensional clothing images without defects as sample labels. In order to enable the fusion model to generate two-dimensional clothing images with as few defects as possible during the model reasoning process, during the model training process, a data enhancement method of randomly adding noise and Mask can be used to construct a batch of data pairs of two-dimensional clothing images. For example, during the reasoning process, for areas where defects are prone to occur in two-dimensional clothing images, some noise is artificially added to construct a data pair including "good data" and "defective data". The defective data is used as sample data and the good data is used as sample labels to train the fusion model.

[0053] Therefore, the naturalness and rationality of the generated two-dimensional clothing images can be greatly improved through the image restoration function of the fusion model.

[0054] After obtaining the two-dimensional clothing image, the two-dimensional clothing image can be input into the back-projection model to generate a target three-dimensional clothing map.

[0055] Specifically, in some examples, the back-projection model can be based on the three-dimensional structure of the 3D character image, and through the back-projection technology, the two-dimensional clothing image can find the correspondence between the mapped UV map color value in the texture space coordinate system and the vertex color of the 3D model mesh, so that the two-dimensional clothing image can be mapped to the three-dimensional clothing surface to obtain the target three-dimensional clothing map.

[0056] Through the method disclosed in the present invention, the dressing of virtual characters will no longer be limited to the pre-configured clothing library, but can generate the target three-dimensional clothing map expected by the user according to the user's creativity and imagination, thereby realizing the personalization and diversification of the virtual character dressing. This can not only enhance the user's interactive experience, but also promote the development of virtual character dressing technology, bringing a wider range of application scenarios and business opportunities to virtual reality applications.

[0057] Figure 2A A flowchart of a non-limiting example process 200 of a method for generating a three-dimensional clothing map according to some embodiments of the present disclosure is shown. As shown in FIG2 , the non-limiting example process 200 includes: obtaining first information from a user (S202) and determining whether the first information is text or an image (S204); when the first information is text, inputting the first information into a text-generated image model (S206); when the first information is an image, inputting the first information into an image-generated image model (S208), thereby generating a clothing front view (S210); then inputting the clothing front view into a second image generation model to generate a clothing back view (S212); inputting the clothing front view and the clothing back view into a fusion model to generate a two-dimensional clothing image (S214); inputting the two-dimensional clothing image into a back projection model to generate a target three-dimensional clothing map (S216).

[0058] Additionally, Figure 2B It is shown without limitation Figure 2A An exemplary implementation of step S208 in FIG. Figure 2B As shown, when it is determined in the aforementioned step S204 that the first information is an image, the first information is input into the image feature extraction model (step S208') to generate an image prompt word (step S218), wherein the data type of the image prompt word is text, which also means that the data type of the first information is converted from image to text; the image prompt word is input into the text graph model (step S220), and then steps S210 to S216 are executed. Here, the text graph model in step S220 can be the text graph model in step S206, that is, the same first image generation model can be used to generate the clothing front view for first information of different data types.

[0059] The present disclosure further provides an electronic device, which may include a processor and a memory storing computer executable instructions, wherein when the computer executable instructions are executed by the processor, the processor executes the method for generating a three-dimensional clothing map according to any of the aforementioned embodiments of the present disclosure. Figure 3 , which shows a schematic block diagram of an electronic device 300 according to some embodiments of the present disclosure. Figure 3 As shown, the electronic device 300 includes a processor 302 and a memory 304 storing computer executable instructions, which, when executed by the processor 302, causes the processor 302 to perform the method for generating a three-dimensional clothing map according to any of the aforementioned embodiments of the present disclosure. The processor 302 may be, for example, a central processing unit (CPU) of the electronic device 300. The processor 302 may be any type of general-purpose processor, or may be a processor specifically designed for generating a three-dimensional clothing map, such as an application-specific integrated circuit ("ASIC"). The memory 304 may be coupled to the processor 302 and may include various computer-readable media accessible by the processor 302. In various embodiments, the memory 304 described herein may include volatile and non-volatile media, removable and non-removable media. For example, the memory 304 may include any combination of the following: random access memory ("RAM"), dynamic RAM ("DRAM"), static RAM ("SRAM"), read-only memory ("ROM"), flash memory, cache memory, and / or any other type of non-transitory computer-readable media. The memory 304 may store instructions that, when executed by the processor 302 , enable the processor 302 to perform the method for generating a three-dimensional clothing map according to any of the aforementioned embodiments of the present disclosure.

[0060] The present disclosure also provides a non-volatile storage medium having computer executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method for modulating a signal according to any of the foregoing embodiments of the present disclosure or the method for generating a three-dimensional clothing map according to any of the foregoing embodiments.

[0061] The present disclosure also provides a computer program product, which may include instructions, and when the instructions are executed by a processor, the method for modulating a signal according to any of the aforementioned embodiments of the present disclosure or the method for generating a three-dimensional clothing map according to any of the aforementioned embodiments may be implemented. The instructions may be any set of instructions to be executed directly by the processor, such as machine code, or any set of instructions to be executed indirectly, such as a script. The instructions may be stored in an object code format for direct processing by the processor, or in any other computer language, including a script or collection of independent source code modules that are interpreted on demand or compiled in advance.

[0062] Figure 4A schematic block diagram of a computer system 400 on which embodiments of the present disclosure may be implemented is shown. The computer system 400 includes a bus 402 or other communication mechanism for transmitting information, and a processing device 404 coupled to the bus 402 for processing information. The computer system 400 also includes a memory 406 coupled to the bus 402 for storing instructions to be executed by the processing device 404, which may be a random access memory (RAM) or other dynamic storage device. The memory 406 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processing device 404. The computer system 400 also includes a read-only memory (ROM) 408 or other static storage device coupled to the bus 402 for storing static information and instructions for the processing device 404. A storage device 410 such as a magnetic disk or optical disk is provided and coupled to the bus 402 for storing information and instructions. The computer system 400 may be coupled to an output device 412 via the bus 402 for providing output to a user, such as, but not limited to, a display (such as a cathode ray tube (CRT) or a liquid crystal display (LCD)), a speaker, etc. An input device 414 such as a keyboard, a mouse, a microphone, etc. is coupled to the bus 402 for transmitting information and command selections to the processing device 404. The computer system 400 may perform embodiments of the present disclosure. Consistent with certain implementations of the present disclosure, the results are provided by the computer system 400 in response to the processing device 404 executing one or more sequences of one or more instructions contained in the memory 406. Such instructions may be read into the memory 406 from another computer-readable medium such as the storage device 410. The execution of the sequence of instructions contained in the memory 406 causes the processing device 404 to perform the methods described herein. Alternatively, hard-wired circuitry may be used instead of software instructions or in combination with software instructions to implement the present teachings. Therefore, implementations of the present disclosure are not limited to any specific combination of hardware circuitry and software. In various embodiments, computer system 400 can be connected to one or more other computer systems such as computer system 400 across a network via network interface 416 to form a networked system. The network can include a private network or a public network such as the Internet. In a networked system, one or more computer systems can store data and supply data to other computer systems. The term "computer-readable medium" as used herein refers to any medium that participates in providing instructions to processing device 404 for execution. Such media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical disks or disks such as storage device 410. Volatile media include dynamic memories such as memory 406. Transmission media include coaxial cables, copper wires, and optical fibers, including wiring that includes bus 402.Common forms of computer-readable media or computer program products include, for example, floppy disks, flexible disks, hard disks, tapes, or any other magnetic media, CD-ROMs, digital video disks (DVDs), Blu-ray disks, any other optical media, thumb drives, memory cards, RAMs, PROMs and EPROMs, flash EPROMs, any other memory chips or boxes, or any other tangible media from which a computer can read. Various forms of computer-readable media may be involved when carrying one or more sequences of one or more instructions to processing device 404 for execution. For example, instructions may initially be carried on a disk of a remote computer. The remote computer may load instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 400 may receive data over a telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector coupled to bus 402 may receive the data carried in the infrared signal and place the data on bus 402. Bus 402 carries the data to memory 406, and processing device 404 retrieves instructions from memory 406 and executes the instructions. Optionally, the instructions received by memory 406 may be stored on storage device 410 either before or after execution by processing device 404 .

[0063] According to various embodiments, instructions configured to be executed by a processing device to perform a method are stored on a computer-readable medium. A computer-readable medium may be a device that stores digital information. For example, a computer-readable medium includes a compact disk read-only memory (CD-ROM) as known in the art for storing software. The computer-readable medium is accessed by a processor adapted to execute the instructions configured to be executed.

[0064] One or more exemplary embodiments of the present disclosure are described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0065] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present disclosure does not exclude that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0066] Although one or more embodiments of the present disclosure provide method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment).

[0067] The terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, product or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, product or apparatus. In the absence of further restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or apparatus comprising the elements. For example, if the words "first", "second" and the like are used to indicate names, they do not indicate any particular order.

[0068] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more embodiments of the present disclosure, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0069] The present disclosure is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0070] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a particular manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device that implements the functions specified in one or more flows of a flowchart and / or one or more blocks of a block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows of a flowchart and / or one or more blocks of a block diagram.

[0071] Those skilled in the art should understand that one or more embodiments of the present disclosure may be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of the present disclosure may be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0072] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0073] The same or similar parts between the various embodiments of the present disclosure can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment. In the description of the present disclosure, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the present disclosure, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in the present disclosure and the features of the different embodiments or examples without contradiction.

[0074] In addition, when used in the present disclosure, the words "herein," "above," "below," "hereunder," "above," and words of similar meaning shall refer to the present disclosure as a whole rather than to any particular portion of the present disclosure. Furthermore, unless expressly stated otherwise or understood otherwise in the context of use, conditional language used herein, such as "may," "might," "for example," "such as," and the like, is generally intended to express that certain embodiments include, while other embodiments do not include, certain features, elements, and / or states. Thus, such conditional language is generally not intended to imply that one or more embodiments require features, elements, and / or states in any way, or whether such features, elements, and / or states are included or performed in any particular embodiment.

[0075] The above description is only an example of one or more embodiments of the present disclosure, and is not intended to limit one or more embodiments of the present disclosure. For those skilled in the art, one or more embodiments of the present disclosure may have various changes and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims.

[0076] In addition, the embodiments of the present disclosure may also include the following examples:

[0077] 1. A method for generating a three-dimensional clothing texture, comprising:

[0078] Acquire first information from a user, where the first information indicates features of a target three-dimensional clothing texture desired by the user;

[0079] determining a data type of the first information, where the data type of the first information includes text and image;

[0080] According to the data type of the first information, the first information is input into a first image generation model to generate a front view of the clothing;

[0081] Inputting the garment front view into a second image generation model to generate a garment back view;

[0082] Inputting the clothing front view and the clothing back view into a fusion model to generate a two-dimensional clothing image;

[0083] The two-dimensional clothing image is input into a back-projection model to generate the target three-dimensional clothing map.

[0084] 2. The method according to item 1, wherein inputting the first information into a first image generation model to generate a front view of a garment comprises:

[0085] The art style features expected by the user are extracted based on the first information, and the art style features and the features of the target three-dimensional clothing map are input into the first image generation model.

[0086] 3. The method according to 2, wherein a style insertion model for adjusting the artistic style of an image is incorporated into the first image generation model as a control weight.

[0087] 4. The method according to 3, wherein, when the data type of the first information is text, the first image generation model includes a diffusion model and a ControlNet model.

[0088] 5. The method according to 4, wherein inputting the first information into the first image generation model to generate the clothing front view comprises:

[0089] The first information is input into the diffusion model, and the parameters of the diffusion model are adjusted based on the ControlNet model and the style plug-in model to generate the clothing front view.

[0090] 6. The method according to 3, wherein:

[0091] When the data type of the first information is an image, the first image generation model includes an image feature extraction model, a diffusion model and a ControlNet model.

[0092] 7. The method according to 6, wherein the image feature extraction model includes a diffusion inversion model.

[0093] 8. The method according to 6, wherein inputting the first information into the first image generation model to generate the clothing front view comprises:

[0094] The first information is input into the image feature extraction model to generate image prompt words.

[0095] 9. The method according to 8, wherein the image prompt words are input into the diffusion model, and the parameters of the diffusion model are adjusted based on the ControlNet model and the style plug-in model to generate the clothing front view.

[0096] 10. The method according to 8, wherein inputting the first information into the first image generation model to generate the clothing front view comprises:

[0097] Performing pose estimation on the image in the first information to determine a deviation between the pose of the image in the first information and the pose of the front view;

[0098] Adjusting the redrawing weight of the ControlNet model based on the deviation between the pose of the image in the first information and the pose of the front view;

[0099] The image prompt words are input into the diffusion model and the parameters of the diffusion model are adjusted based on the ControlNet model and the style plug-in model to generate the clothing front view.

[0100] 11. The method according to 10, wherein:

[0101] In response to a deviation between a pose of the image in the first information and a pose of the front view being less than 30°, adjusting a redrawing weight of the ControlNet model to 0.3;

[0102] In response to a deviation between a pose of the image in the first information and a pose of the front view being not less than 30° and not more than 60°, adjusting a redrawing weight of the ControlNet model to 0.5;

[0103] In response to a deviation between the pose of the image in the first information and the pose of the front view being greater than 60°, adjusting the redrawing weight of the ControlNet model to 0.8.

[0104] 12. The method according to item 1, wherein the fusion model comprises a generative adversarial network for image restoration, and inputting the clothing front view and the clothing back view into the fusion model to generate a two-dimensional clothing image comprises:

[0105] Inputting the clothing front view and the clothing back view into the fusion model to generate an initial two-dimensional clothing image;

[0106] Detecting defects in the initial two-dimensional clothing image;

[0107] In response to detecting that the initial two-dimensional clothing image has defects, image restoration is performed on the initial two-dimensional clothing image based on the generative adversarial network to obtain the two-dimensional clothing image.

[0108] 13. The method according to 3, wherein the style insertion model is trained using first training data, wherein the first training data includes a plurality of sample clothing images having a preset art style.

[0109] 14. The method according to 13, wherein the first training data trains the style insertion model in a low-rank adaptive manner.

[0110] 15. The method according to 1, wherein the second image generation model is trained using second training data, the second training data comprising a plurality of front views of clothing as samples and a corresponding plurality of back views of clothing as labels of the samples.

[0111] 16. The method according to 1, wherein the fusion model is trained using third training data, the third training data comprising a plurality of two-dimensional clothing images with defects as samples and the plurality of two-dimensional clothing images without defects as labels of the samples.

[0112] 17. An electronic device comprising:

[0113] Processor; and

[0114] A memory storing computer executable instructions, which, when executed by the processor, cause the processor to perform the method for generating a three-dimensional clothing map according to any one of 1 to 16.

[0115] 18. A non-transitory storage medium having computer executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the method for generating a three-dimensional clothing map according to any one of 1 to 16.

[0116] 19. A computer program product, comprising instructions, which, when executed by a processor, implement the method for generating a three-dimensional clothing map according to any one of 1 to 16.

Claims

1. A method for generating a three-dimensional clothing map, comprising: Acquire first information from a user, where the first information indicates features of a target three-dimensional clothing texture desired by the user; determining a data type of the first information, where the data type of the first information includes text and image; inputting the first information into a first image generation model to generate a front view of the garment according to a data type of the first information; Inputting the garment front view into a second image generation model to generate a garment back view; Inputting the clothing front view and the clothing back view into a fusion model to generate a two-dimensional clothing image; The two-dimensional clothing image is input into a back-projection model to generate the target three-dimensional clothing map.

2. The method according to claim 1, wherein: Inputting the first information into a first image generation model to generate a front view of the clothing includes: extracting the art style features expected by the user based on the first information, and inputting the art style features and the features of the target three-dimensional clothing map into the first image generation model, Among them, the style insertion model for adjusting the artistic style of the image is integrated into the first image generation model as a control weight.

3. The method according to claim 2, wherein: When the data type of the first information is text, the first image generation model includes a diffusion model and a ControlNet model. Inputting the first information into the first image generation model to generate the clothing front view comprises: Inputting the first information into the diffusion model, and adjusting the parameters of the diffusion model based on the ControlNet model and the style plug-in model to generate the clothing front view; When the data type of the first information is an image, the first image generation model includes an image feature extraction model, a diffusion model, and a ControlNet model, wherein the image feature extraction model includes a diffusion inversion model. Inputting the first information into the first image generation model to generate the clothing front view comprises: Inputting the first information into the image feature extraction model to generate image prompt words, The image prompt words are input into the diffusion model, and the parameters of the diffusion model are adjusted based on the ControlNet model and the style plug-in model to generate the clothing front view.

4. The method according to claim 3, wherein: Inputting the first information into the first image generation model to generate the clothing front view comprises: Performing pose estimation on the image in the first information to determine a deviation between the pose of the image in the first information and the pose of the front view; Adjusting the redrawing weight of the ControlNet model based on the deviation between the pose of the image in the first information and the pose of the front view; The image prompt words are input into the diffusion model and the parameters of the diffusion model are adjusted based on the ControlNet model and the style plug-in model to generate the clothing front view.

5. The method according to claim 4, wherein: In response to a deviation between a pose of the image in the first information and a pose of the front view being less than 30°, adjusting a redrawing weight of the ControlNet model to 0.3; In response to a deviation between a pose of the image in the first information and a pose of the front view being not less than 30° and not more than 60°, adjusting a redrawing weight of the ControlNet model to 0.5; In response to a deviation between the pose of the image in the first information and the pose of the front view being greater than 60°, adjusting the redrawing weight of the ControlNet model to 0.

8.

6. The method according to claim 1, wherein: The fusion model includes a generative adversarial network for image restoration, and inputting the clothing front view and the clothing back view into the fusion model to generate a two-dimensional clothing image includes: Inputting the clothing front view and the clothing back view into the fusion model to generate an initial two-dimensional clothing image; Detecting defects in the initial two-dimensional clothing image; In response to detecting that the initial two-dimensional clothing image has defects, image restoration is performed on the initial two-dimensional clothing image based on the generative adversarial network to obtain the two-dimensional clothing image.

7. The method according to claim 2, wherein: Using first training data to train the style insertion model, the first training data comprising a plurality of sample clothing images having a preset art style, wherein the first training data trains the style insertion model in a low-rank adaptation manner; Training the second image generation model using second training data, the second training data comprising a plurality of front views of clothing as samples and a corresponding plurality of back views of clothing as labels of the samples; The fusion model is trained using third training data, wherein the third training data includes a plurality of two-dimensional clothing images with defects as samples and the plurality of two-dimensional clothing images without defects as labels of the samples.

8. An electronic device, comprising: processor; as well as A memory storing computer executable instructions, which, when executed by the processor, cause the processor to perform the method for generating a three-dimensional garment map according to any one of claims 1 to 7.

9. A non-transitory storage medium having computer executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the method for generating a three-dimensional garment map according to any one of claims 1 to 7.

10. A computer program product, comprising instructions, which, when executed by a processor, implement the method for generating a three-dimensional garment map according to any one of claims 1 to 7.

Images