Garment multi-view generation method and device

Abstract

The invention relates to an innovative garment multi-view-angle generation method and device. Multi-view-angle images with high quality and consistency are generated through a self-adaptive local condition multi-view diffusion model. Screening and acquiring a large number of model clothing image data sets from the clothing database, and combining the data sets into clothing image pairs after screening; global and local features are dynamically selected by using a self-adaptive condition feature selection module so as to adapt to requirements of different visual angles; a united attention-based diffusion UNet network is constructed, and a cross-view interaction self-attention module and a cascade cross attention module are adopted to enhance the consistency and quality of multi-view images; the model is optimized and trained through a joint loss function, and the high quality of the generated image at the pixel and visual perception level is ensured. According to the method, the limitation of the traditional technology is broken through, the accuracy and consistency of garment view generation are remarkably improved, the innovative application in the field of garment design is promoted, and the satisfaction degree of consumers on garment display is improved.

Description

Technical Field

[0001] The invention belongs to the field of computer and network technology, and particularly relates to a method and device for generating clothing from multiple viewing angles. Background Art

[0002] In the field of clothing multi-view generation technology, despite the progress of traditional image processing and 3D modeling technology, there are still limitations in generating high-quality, personalized and consistent multi-view clothing views. Traditional multi-view synthesis methods mainly rely on visual reconstruction technology, but when the training data coverage is insufficient or the character poses are inconsistent, the generated clothing views are often inaccurate and inconsistent. In addition, the overfitting problem under a fixed view may cause image distortion and blur, limiting its application in high-quality and diverse clothing view generation.

[0003] Although diffusion models perform well in conditional image generation and simple 3D object generation, factors such as illumination changes, pose changes, and background interference make it challenging to maintain consistency and detail texture stability when dealing with multi-view generation. Existing methods usually ignore the complex spatial relationship and occlusion problems between clothing and models, resulting in the generated clothing images that may not be realistic enough to meet high display requirements. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention proposes a method and device for generating clothing from multiple perspectives.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] On the one hand, the present invention provides a method for generating clothing from multiple perspectives, comprising the following steps:

[0007] S1: Obtain a model clothing image dataset from the clothing database. The dataset includes more than 40,000 model clothing objects. For each object, select its front view, back view, left view and right view and the corresponding camera view parameters to form a clothing image pair. The front view is specified as the input conditional image, and the other views are used as the real label images to be predicted. They are divided into training set, validation set and test set according to 8:1:1;

[0008] S2: The user manually uploads the model clothing image that needs to be generated from multiple perspectives in the system. The system will automatically segment the clothing in the model clothing image using the pre-trained SAM model to obtain the clothing posture image, and use OpenPose to extract the skeleton images of the model clothing image as posture information and store them together in the system for subsequent operations;

[0009] S3: Design an adaptive local conditional multi-view diffusion model (ALC-MVD) to achieve multi-view generation of clothing model images. The ALC-MVD model includes a clothing view adaptive conditional feature selection module (ACFS-GMV) and a joint attention-based diffusion UNet network (JA-UDM), and uses a joint loss function to optimize the overall network training;

[0010] S4: Input the clothing image pair into the ALC-MVD model. First, segment the input front view using the pre-trained SAM model to obtain the front view clothing posture map. Then, input the clothing posture map into the clothing view adaptive conditional feature selection module to extract global and local conditional features using the hard conditional feature selection module and the soft conditional feature selection module respectively. Next, input the extracted conditional features into the diffusion UNet network based on joint attention. Use the cross-view interactive self-attention module (CVI-SA) and the cascaded cross attention module (C-CA) to fuse and generate diffusion probabilities for the original input multi-view model clothing image and the conditional features, and output the corresponding multi-view generated image. Finally, the output multi-view generated image and the original input multi-view model clothing image are used to calculate the loss value through the joint loss function, and the back propagation and gradient descent algorithms are used to optimize and train the parameters of some learnable modules of the ALC-MVD model.

[0011] S5: Load the ALC-MVD model weights and network structure trained in S4, obtain the model clothing image and the corresponding clothing posture image input by the user, and input them into the loaded ALC-MVD model together;

[0012] S6: In the ALC-MVD model reasoning process, the model clothing image and clothing posture image are used as conditions, and the ALC-MVD model is used to gradually estimate the denoised multi-view generation result from the pure noise, and the multi-view result of the corresponding model clothing image is output;

[0013] Preferably, the specific operation steps for generating clothing image pairs in step S1 are:

[0014] S11: Filter model clothing images with the term “model” from the existing clothing database to form a model clothing image dataset;

[0015] S12: Perform fine screening on the model clothing image dataset, retaining only the model clothing objects covering the front view, back view, left view and right view, and discarding other incomplete model clothing objects;

[0016] S13: using Opencv library functions to quickly calibrate camera parameters on the front view of the model's clothing to obtain corresponding camera parameters;

[0017] S14: The front view, back view, left view and right view of the above model clothing and their camera parameters are combined into a clothing image pair, and the front view is specified as the input conditional image, and the other views are used as the real label images that need to be predicted, and then divided into a training set, a validation set and a test set according to 8:1:1.

[0018] Preferably, the method of the adaptive condition feature selection module (ACFS-GMV) in step S3 is:

[0019] The adaptive conditional feature selection module (ACFS-GMV) aims to improve the consistency and quality of multi-view generated images. The module adaptively selects the features of clothing views as conditional features to adapt to the specific needs of clothing under different perspectives. ACFS-GMV includes a hard conditional feature selection module, a soft conditional feature selection module and a multi-scale feature extraction module, which respectively realize the extraction of global clothing features, local clothing features and multi-scale clothing features. By utilizing the above modules, the ACFS-GMV module can adaptively and dynamically adjust the feature selection strategy according to the similarity between the input clothing model image and the corresponding clothing posture image. This adaptability enables the model to better cope with the clothing feature extraction requirements under different perspectives and postures. The model retains the global clothing semantic information while achieving more accurate texture detail modeling, thereby improving the accuracy and consistency of the generated images.

[0020] Preferably, the method of the joint attention-based diffusion UNet network (JA-UDM) described in step S3 is:

[0021] The joint attention-based diffusion UNet network (JA-UDM) structure and the UNet architecture in the traditional SDXL model adopt a downsampling-upsampling framework, in which the image resolution is gradually reduced in downsampling to extract features, and the image resolution is gradually restored in upsampling to generate multi-view clothing images. Unlike the traditional SDXL model, JA-UDM replaces the self-attention method and the cross-attention method in the upsampling process by using the cross-view interactive self-attention module (CVI-SA) and the cascaded cross-attention module (C-CA). The consistency and quality in multi-view images are effectively enhanced by using the form of joint attention, and the lost and erroneous texture details are refined through selective fusion, and the ability to retain high-frequency details in the original clothing is strengthened. At the same time, JA-UDM retains the iterative denoising ability of the diffusion model, converts the Gaussian distribution into the target distribution, and realizes the generation of high-quality multi-view clothing images from random Gaussian noise.

[0022] Preferably, the joint loss function described in step S3 is:

[0023] The ALC-MVD model uses a joint loss function for optimization training, including diffusion loss function, reconstruction loss function and perceptual loss function, to ensure the high quality of the generated image at the pixel level and perceptual level. LDM Aims to minimize the difference between the predicted noise and the actual noise to improve the quality of the generated image; the reconstruction loss L1 calculates the difference between the model-generated image and the original input image by calculating the first norm as an indicator of image consistency; the perceptual loss L perc Focusing on the quality of the generated image in visual perception, it is usually calculated by comparing the difference between the model-generated image and the original image at the feature level. In this paper, the pre-trained VGG network is selected to extract deep features and calculate the difference between feature vectors, and the weights λ1 and λ perc The best balance point of the model is found by choosing to achieve the best image generation effect. During the model training process, the parameters of the ALC-MVD model are optimized by minimizing the joint loss function to ensure that when the model generates multi-view clothing images, it is not only consistent with the original image at the pixel level, but also close to the real image in visual perception.

[0024] Preferably, the method of the hard condition feature selection module in step S4 is:

[0025] The hard condition feature selection module is a key component of the adaptive local conditional multi-view diffusion model (ALC-MVD), which is responsible for extracting global features from clothing images to support the multi-view generation process. First, the model clothing image is segmented by the pre-trained SAM model to obtain more accurate clothing regions; then, the hard condition feature selection module will evaluate the similarity between the clothing pose and the model pose, which helps to determine which view of the clothing image is most suitable as a representative of the global features; then, based on the evaluation results of the pose similarity, the module will perform hard selection, that is, select the clothing image closest to the target model pose to ensure that the extracted clothing global features are highly consistent with the target view; next, the CLIP-based image encoder is used to extract the global features of the clothing image to ensure that the important semantic information of the clothing can be captured; finally, the segmented clothing image is input into the image encoder for global feature encoding and input into the UNet model of JA-UDM as a hard condition to guide the image reconstruction in the multi-view generation process. The features extracted by the hard conditional feature selection module can ensure that the global features of the clothing remain consistent in each view generated by multi-view, thereby improving the quality and realism of the generated images.

[0026] Preferably, the method of the soft condition feature selection module in step S4 is:

[0027] The soft conditional feature selection module is a key component for extracting and fusing local features in the adaptive local conditional multi-view diffusion model (ALC-MVD). This module uses a careful feature selection and fusion strategy to enhance the detail expression and realism of multi-view clothing image generation. First, the pre-trained SAM model is used to segment the clothing of the model clothing image in the front view to obtain more accurate clothing regions; then, a trainable multi-scale feature encoder E is designed. l (), extract multi-scale local features from the clothing image of the front view, these features contain clothing texture, pattern and other detailed information; then, use OpenPose technology to extract skeleton images from the model clothing image of the front view, these skeleton images provide key information of the character posture, and through a trainable posture encoder, the extracted skeleton images are converted into posture features; next, the extracted posture features are combined with the multi-scale clothing local features, and the weight adjustment mechanism in the soft conditional feature selection module is used to selectively enhance or weaken the features of different views; finally, according to the pose similarity and the geometric relationship between views, the fused local features are mapped to the corresponding UNet network scale in JA-UDM as soft condition input to assist the generation of multi-view images. Different from the deterministic selection of the hard conditional feature selection module, the soft conditional feature selection module adopts a soft selection mechanism to weight the clothing features of different views to meet the needs of multi-view reconstruction. This weighting strategy provides rich local feature support for the ALC-MVD model, allowing the model to flexibly adjust the importance of features in different situations, making the generated multi-view clothing images richer and more realistic in details.

[0028] Preferably, the method of the cross-view interactive self-attention module (CVI-SA) in step S4 is:

[0029] The cross-view interactive self-attention module (CVI-SA) promotes information interaction and fusion between different views by establishing associations between multi-view image features, thereby achieving multi-dimensional feature representation and detail enhancement. Firstly, a position encoding technique inspired by the light field network (LFN) is used to associate a ray with each pixel in the CVI-SA module and represent it using Plücker coordinates. Strict geometric constraints and camera parameters are introduced to achieve accurate pixel-level feature encoding. Then, an adjacent view cross-attention module is designed to aggregate ray features from adjacent views of the target view. Interaction across views is achieved through position encoding and projection features, and feature correlations between different views are captured. Next, a novel ray self-attention module is used to simulate the contribution of spatial points to pixel-level features. The deep weights of sampled rays are modeled through the self-attention mechanism to achieve ray aggregation, revealing the deep relationship and spatial correlation between sampled points along the ray, and merging spatial point features into pixel features. Finally, a weighted summation method is used to fuse geometric spatial features into pixel features of the target unit block to achieve efficient multi-view. Figure 1 Consistency modeling. Through the CVI-SA module, the ALC-MVD model can achieve high-quality generation of multi-view clothing images while maintaining consistency between perspectives and richness of details.

[0030] Preferably, the method of cascading the cross attention module (C-CA) in step S4 is:

[0031] The cascaded cross attention module (C-CA) deeply fuses global features with local features through attention cascade, which is used to further enhance the details and consistency of multi-view clothing image generation. First, the C-CA module performs self-attention calculation on the current feature to capture the dependency in the local area and enhance the expressiveness of the local feature; then, based on the self-attention, a double cross attention mechanism is introduced, the currently processed feature in the module is used as the query, the global feature and the local feature are used as the key and value respectively, and a learnable fusion vector λ is designed. c , selectively fuses global and local features in the channel dimension, strengthens the retention of high-frequency details, and retains global semantic information; next, uses local features to refine and correct global features, so that the C-CA module can enhance details in clothing images, such as textures and patterns, and improve the realism of generated images; finally, by operating at different scales, it ensures that fine alignment and deep fusion of features can be achieved at all scales. Through the C-CA module, the ALC-MVD model can achieve more refined and consistent detail representation in multi-view clothing image generation, improving the overall generation effect.

[0032] Preferably, the method of performing reasoning using the ALC-MVD model according to the model clothing picture input by the user in step S6 is:

[0033] S61: Segment the model clothing image input by the user through the pre-trained SAM model to obtain a clothing posture image;

[0034] S62: Input the model clothing image into the image encoder to encode the latent space features, and input it into the diffusion UNet network based on joint attention;

[0035] S63: Calculate the camera parameters corresponding to the clothing posture graph and input them into the adaptive condition feature selection module;

[0036] S64: The global and multi-scale local conditional features output by the adaptive conditional feature selection module are input into the diffusion UNet network based on joint attention, and the alignment and fusion of the current features and the conditional features are realized through the cross-view interactive self-attention module (CVI-SA) and the cascaded cross attention module (C-CA);

[0037] S65: Using the fused features, the joint attention-based diffusion UNet network starts from pure noise and gradually denoises it to restore a clear image from the random state to generate multi-view clothing images;

[0038] S66: The system outputs multi-view clothing images generated by the ALC-MVD model to meet the user's needs for clothing display at different perspectives.

[0039] On the other hand, the present invention also provides a clothing multi-view image generation device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, the steps of the clothing multi-view generation method of the above-mentioned ALC-MVD model are implemented.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention enhances the generation consistency of model clothing images under different viewing angles through the adaptive local conditional multi-view diffusion model (ALC-MVD), and solves the limitations of traditional methods in multi-view reconstruction.

[0042] Secondly, the clothing view adaptive conditional feature selection module (ACFS-GMV) can use hard conditional feature selection and soft conditional feature selection modules to adaptively extract global and local features according to the clothing and model posture information, which improves the generalization ability and training efficiency of the model; then, the joint attention-based diffusion UNet network (JA-UDM) effectively enhances the retention of high-frequency details in clothing images through the cross-view interactive self-attention module (CVI-SA) and the cascaded cross attention module (C-CA), improving the detail quality and realism of the image;

[0043] Finally, a joint loss function combining diffusion loss, reconstruction loss and perceptual loss is designed to ensure the high quality of generated images at the pixel level and in visual perception.

[0044] The present invention significantly improves the realism and consistency of clothing multi-view generation through an innovative multi-perspective generation method, which not only provides consumers with a more intuitive and detailed clothing display and enhances their shopping experience, but also significantly reduces the time cost of clothing design and shortens the product cycle from design to market, thereby promoting the digital transformation and global development of the clothing design industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a logic flow chart of the present invention;

[0046] Figure 2 It is the ALC-MVD network training structure diagram of the present invention;

[0047] Figure 3 It is the ALC-MVD network reasoning structure diagram of the present invention;

[0048] Figure 4 It is a structural diagram of the CVI-SA module in the JA-UDM network of the present invention;

[0049] Figure 5 It is a structural diagram of the C-CA module in the JA-UDM network of the present invention;

[0050] Figure 6 It is a structural diagram of the soft condition selection module of the present invention;

[0051] Figure 7 This is a display diagram of the multi-view generation results of the present invention. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0053] like Figure 1-7As shown, the embodiment provides a method for generating clothing from multiple perspectives.

[0054] A model clothing image dataset is obtained from the clothing database, which includes more than 40,000 model clothing objects. For each object, its front view, back view, left view and right view and the corresponding camera perspective parameters are selected to form a clothing image pair, and then divided into training set, validation set and test set according to 8:1:1.

[0055] The user manually uploads the model clothing image that needs to be generated from multiple perspectives in the system. The system will automatically segment the clothing in the model clothing image using the pre-trained SAM model to obtain clothing images and store them in the system for subsequent operations.

[0056] An adaptive local conditional multi-view diffusion model (ALC-MVD) is established to realize multi-view generation of clothing model images. The model mainly consists of a clothing view adaptive conditional feature selection module and a diffusion UNet network based on joint attention, which respectively realizes the conditional feature generation of related clothing views and the diffusion generation of multi-view model clothing images, and uses a joint loss function to optimize the overall network training.

[0057] Many advanced multi-view synthesis methods are based on visual reconstruction, but they have limitations in generating high-quality personalized clothing views, especially when the training data coverage is insufficient, the pose inconsistency of the target person may occur. In contrast, diffusion model-based methods can achieve excellent performance in the fields of image generation and simple 3D object generation due to their powerful conditional image generation capabilities. However, the versatility of these methods is still limited, and it is difficult to simultaneously generate multiple clothing views with consistent perspectives in clothing design scenarios. To improve multi-view Figure 1 In order to improve the consistency of the image quality, researchers have tried to introduce 3D features constructed from global multi-view images into the diffusion model. However, this method not only slows down the generation process, but also easily overfits on a fixed viewing angle, resulting in distortion and blurring of the image. To this end, the present invention proposes a method for adaptive selection of clothing view conditional features, which selects adaptive features of clothing in different views based on the posture information of people and clothing. Among them, the hard selection module selects one of the four different view clothing for global feature extraction, and the soft selection module modulates and selects the four view clothing for local feature extraction. By using the feature map of the adjacent view with less occlusion as the conditional feature to generate the target view image, the overall generalization ability and training efficiency of the model can be effectively improved. In addition, in order to enhance the retention of high-frequency details in clothing, a joint attention mechanism is designed to independently align global and local features with character features, and selectively fuse them to refine local clothing details while retaining global semantic information.

[0058] The clothing view adaptive conditional feature selection module (ACFS-GMV) implemented in the ALC-MVD model is:

[0059] In the clothing multi-view generation task, given the significant differences between the front and back views of clothing, it is necessary to extract and assign clothing features of the front and back views of the model in a targeted manner. Existing studies have shown that by determining which view's clothing features to focus on during the fitting process based on the target model's posture. For example, if the target model's posture is a side view, then it is obvious that more reliance should be placed on the features of the back view clothing to generate the fitting results. Specifically, the present invention proposes a clothing view adaptive conditional feature selection module to achieve this function, including a hard conditional feature selection module and a soft conditional feature selection module.

[0060] For the hard condition feature selection module, the present invention adopts the CLIP-based image encoder E M () is used to extract the global features of clothing. Before this process, it is necessary to hard select the clothing images in the front and back views based on the similarity between the clothing pose and the model pose. This means that only one piece of clothing that is closest to the clothing model pose needs to be selected as the input of the image encoder, because it is sufficient to cover the global semantic information. To this end, the pre-trained SAM model is selected to segment the model clothing image with the highest similarity to obtain the clothing image containing the most clothing semantic information.

[0061] For the soft conditional feature selection module, first, a trainable image feature encoder E l () Extract multi-scale local features of clothing in the front view. However, simply combining view conditions with clothing features may lead to functional confusion. To this end, the present invention introduces a soft selection mechanism to modulate their local features separately. Next, the skeleton image obtained from the model clothing view is used as posture information, and the front view model skeleton posture is defined as p h Then, a trainable pose encoder E is used p () is used to extract the features E of each posture image p (p h ). p () is a micro network consisting of three blocks and a layer normalization operation. Each block contains a convolutional layer, a GELU activation layer, and a downsampling operation. Finally, the encoded posture embedding vector, the model clothing graph features, and the multi-scale clothing posture graph features are used as the input of the proposed soft conditional feature selection module to obtain the features of each scale.

[0062] For example, when processing front view clothing, in the i-th soft conditional feature selection module, the weight is The linear layer encodes the pose of the model skeleton in the front view E p (p h ) Model pose embedding mapped to the front view Then use the image feature encoder E l () Encoding the front view clothing pose graph features Features of the model clothing image with the corresponding front view and pass it through with weight The linear layer maps to the front view clothing embedding Mannequin garment embed with front view Then calculate the similarity between the pose of the model's clothing and the pose of the front clothing, and get the selection weight of the front clothing, that is:

[0063]

[0064] Among them, weights represents the selection weight of the front view clothing, and d represents the dimension of these matrices. According to the model pose benchmark, if it is front view, the similarity between the model pose and the front view clothing pose in the result will be higher. Therefore, the corresponding front view clothing features will be enhanced by weights, and vice versa. Finally, the posture clothing features of different scales and the weighted front view clothing features are connected along the channel dimension as local conditions of different scales in the UNet network.

[0065] In summary, the present invention performs hard selection when extracting global features and soft selection when extracting local features based on the similarity between the input clothing model image and the corresponding clothing posture image. In order to use local features to enhance the high-frequency details in the global features while retaining the semantic information in the clothing, the global clothing features c are extracted by the hard condition feature selection module. g Provide the same conditions for each scale upsampling block in UNet and use the soft conditional feature selection module to capture multi-scale local clothing features to allow reconstruction of more accurate details.

[0066] The joint attention-based diffusion UNet network (JA-UDM) implemented in the ALC-MVD model is:

[0067] Diffusion models have shown strong capabilities in visual generation, which transform Gaussian distribution into target distribution through iterative denoising. In particular, Stable Diffusion is a widely used generative diffusion model, which is composed of CLIP text encoder E T (), VAE encoder E() and decoder D() and conditional denoising model ∈ θ Composition. Text Encoder E T() encodes the input text prompt y as a conditional input. The VAE encoder E() encodes the input image z x Compressed into the latent space to obtain the latent variable z0 = E(z x ). In contrast, the VAE decoder D() decodes the output of the backbone from the latent space to the pixel space. The forward process is performed at any time step t through the VAE encoder E(). Among them, ε~N(0,1) is a random Gaussian noise, and β is a predefined variance table. The training goal of the diffusion model is to obtain a noise prediction network that can minimize the difference between the predicted noise and the real noise added to the latent space features. The diffusion loss function can be defined as:

[0068]

[0069] Where t is the time step, z t In the present invention, the SDXL model is used as the backbone, and the image c is used instead of the text as the conditional input, and then the image encoder E of CLIP is used. M () encodes c, while retaining the attention and convolution operations of the downsampling part of UNet, and replacing the self-attention and cross-attention operations of the upsampling part with the cross-view interactive self-attention module (CVI-SA) and the cascaded cross-attention module (C-CA), respectively. Therefore, the diffusion loss function in the above equation can be modified as follows:

[0070]

[0071] The cross-view interactive self-attention module (CVI-SA) is implemented in the joint attention-based diffusion UNet network (JA-UDM) as follows:

[0072] In order to improve the consistency, quality and efficiency of multi-view generation, a local interactive multi-view diffusion scheme is proposed based on the existing diffusion model. A lightweight cross-view interactive attention block is designed in the original feature space of the diffusion model to establish the association of neighboring views to achieve a multi-dimensional representation of the model. By interacting between multi-view image features, single-view synthesis is extended to multi-view synthesis.

[0073] Specifically, we design a cross-view interactive self-attention module (CVI-SA) and insert it into the upsampling process of the SDXL diffusion model to replace the original self-attention operation. The CVI-SA module leverages the light constraints in stereo vision to enhance the consistency and quality in multi-view images by promoting information exchange between latent features of adjacent views. The CVI-SA module mainly consists of two attention blocks supplemented by linear layers: (a) an adjacent view cross-attention module, whose task is to aggregate light features from adjacent views to obtain light features of the current view; (b) a light self-attention module, which aims to model the depth weights of sampled rays and fuse spatial points into pixel features.

[0074] In order to enhance the module's understanding of perspective relationships, the present invention adopts a position encoding technique inspired by the light field network (LFN). The uniqueness of this method is that each pixel is associated with a ray and represented by Plüücker coordinates, denoted as r = (o*d, d), where o represents the origin of the ray and d represents the direction of the ray. Unlike the traditional position encoding method in which all pixels share the same position encoding in the image method, the LFN method uses the MVP transformation to back-project each pixel in the image into the global coordinate system of the image, accurately locates the light corresponding to each pixel, and effectively applies the encoding at the pixel level by introducing strict geometric constraints and fully combining the camera parameters; then the harmonic transformation is used to integrate these encoded coordinates obtained by LFN with the potential feature map to significantly enrich the spatial information in the feature representation. In addition, in order to solve the potential deviation caused by the use of absolute position, a GPNR-based strategy is used to establish a relative light positioning system using the corresponding camera extrinsics [R, T] of the clothing model image, and the Y-axis R(y) and the light direction (v) are used as the Z-axis to construct the normalized transformation T. Let And y′=y-(y*v′)v′, then the transformation T is defined as:

[0075]

[0076] T=[R c T |-R c T T]

[0077] By applying T to each ray, they are adjusted to originate from (0, 0, 0) and align along the direction (0, 0, 1). Finally, all rays from neighboring viewpoints are transformed into the relative image coordinate system to ensure that the rays have a consistent representation relative to a common reference, thereby mitigating the bias introduced by absolute positioning.

[0078] For the adjacent view cross attention module, given a target view, the output feature map is generated by relying on the features of the F closest model person views in four different views. First, rays are emitted from the cell block on the target view and S points are uniformly sampled along each ray. Then, they are projected onto F-1 adjacent views. For each point s, the projected features {f i s |1≤i≤F-1} to get the feature v s Finally, a cross-view interaction is achieved through a cross-attention layer. Specifically, the point feature v is used s As a query, implement Attention calculation, where:

[0079]

[0080] Where [·] represents a series operation, W Q , W K , W V are all learnable weight matrices that project the input onto the query, key, and value, respectively. In the synthesis process of the target view feature map, the features of nearby views that have a geometric relationship with the target view will be integrated, thereby effectively capturing the feature correlation between different views and effectively enhancing the multi-view Figure 1 Consistency.

[0081] For the light self-attention module, in order to simulate the contribution of spatial points to pixel-level features, self-attention is used to achieve light aggregation. First, get each sampling point characteristics, and from All query, key, and value features are derived from the ray, and self-attention is applied along the S-dimensional rays. This process is crucial for revealing the deep relationship between the sampled points along the ray and capturing their spatial correlation. Finally, the geometric spatial features are fused into the pixel features of the target unit block through a weighted summation method.

[0082] By using ray projection to accurately align features within four resolution scales in the UNet latent space, the feature integrity of the base model is preserved, further achieving efficient multi-view Figure 1 Consistent modeling.

[0083] Step S7, the cascaded cross attention module (C-CA) is implemented in the joint attention-based diffuse UNet network (JA-UDM) as follows:

[0084] After the cross-view interactive self-attention module, the present invention connects a novel cascade cross-attention module to integrate the global feature c g and local features c l In addition, in order to preserve the global features cg Most of the semantic information in the l To refine some lost and erroneous texture details, and selectively merge c g Information in.

[0085] Specifically, in the i-th cascade cross attention module, first, the current feature is calculated of self-attention followed by double cross-attention, where the query (Q) comes from and the global feature c g As a set of keys (K) and values ​​(V), the local features Serves as another set of keys (K) and values ​​(V). After aligning the pose of the model character through cross attention, the clothing feature c g and Selectively fuse on the channel dimension, i.e.:

[0086]

[0087] in Represent the query Q, key K, and value V vectors of the global branch respectively, Represent the query Q, key K, and value V vector of the local branch, respectively. c is a learnable fusion vector, o represents matrix multiplication in the channel dimension, Represents the clothing features after selective fusion. By incorporating and fusing the global and local features of clothing, the ability to retain high-frequency details such as text and patterns in the original clothing can be significantly enhanced.

[0088] The method for implementing the joint loss function for model training in the ALC-MVD model is:

[0089] The diffusion model learns to generate images from random Gaussian noise. However, the above diffusion loss function L LDM The training objective of is performed in the latent space, and the generated results are not explicitly constrained in the visible image space, resulting in slightly different image colors from the actual situation. To alleviate this problem, the present invention introduces L1 loss and perceptual loss L perc , to improve the quality and multi-view consistency of the generated images. The calculation formula of L1 loss is as follows:

[0090]

[0091] in, is the image reconstructed using the ALC-MVD model, and x is the original input image. The perceptual loss calculation formula is as follows:

[0092]

[0093] in, represents the kth layer of the VGG model. In general, the joint loss function can be defined as:

[0094] L=L LDM +λ1L1+λ perc L perc

[0095] Among them, λ1 and λ perc is the weight value used to balance different loss functions.

[0096] The goal of network training is to minimize the network joint loss function.

[0097] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for generating clothing from multiple perspectives, characterized in that: The following steps are involved: S1: Obtain a model clothing image dataset from the clothing database. The dataset includes more than 40,000 model clothing objects. For each object, select its front view, back view, left view and right view and the corresponding camera view parameters to form a clothing image pair. The front view is specified as the input conditional image, and the other views are used as the real label images to be predicted. They are divided into training set, validation set and test set according to 8:1:1; S2: The user manually uploads the model clothing image that needs to be generated from multiple perspectives in the system. The system will automatically segment the clothing in the model clothing image using the pre-trained SAM model to obtain the clothing posture image, and use OpenPose to extract the skeleton images of the model clothing image as posture information and store them together in the system for subsequent operations; S3: Design an adaptive local conditional multi-view diffusion model to realize multi-view generation of clothing model images. The adaptive local conditional multi-view diffusion model includes a clothing view adaptive conditional feature selection module and a diffusion UNet network based on joint attention, and uses a joint loss function to optimize the overall network training; S4: Input the clothing image pair into the adaptive local conditional multi-view diffusion model. First, segment the input front view using the pre-trained SAM model to obtain the front view clothing posture map. Then, input the clothing posture map into the clothing view adaptive conditional feature selection module, and extract the global and local conditional features using the hard conditional feature selection module and the soft conditional feature selection module respectively. Next, input the extracted conditional features into the diffusion UNet network based on joint attention, and use the cross-view interactive self-attention module and the cascade cross-attention module to fuse the original input multi-view model clothing image and the conditional features and generate diffusion probabilities, and output the corresponding multi-view generated image. Finally, the output multi-view generated image and the original input multi-view model clothing image are used to calculate the loss value through the joint loss function, and the back propagation and gradient descent algorithms are used to optimize and train the parameters of some learnable modules of the adaptive local conditional multi-view diffusion model. S5: Load the weights and network structure of the adaptive local conditional multi-view diffusion model trained in S4, obtain the model clothing image and the corresponding clothing posture image input by the user, and input them together into the loaded adaptive local conditional multi-view diffusion model; S6: In the process of inference of the adaptive local conditional multi-view diffusion model, the model clothing image, skeleton image and clothing posture image are taken as input, and the adaptive local conditional multi-view diffusion model is used to gradually estimate the denoised multi-view generation result from pure noise, and output the multi-view result of the corresponding model clothing image.

2. A method for generating clothing from multiple perspectives according to claim 1, characterized in that: The specific operation steps for generating clothing image pairs in step S1 are: S11: Filter model clothing images with the term "model" from the existing clothing database to form a model clothing image dataset; S12: Perform fine screening on the model clothing image dataset, retaining only the model clothing objects covering the front view, back view, left view and right view, and discarding other incomplete model clothing objects; S13: using Opencv library functions to quickly calibrate camera parameters on the front view of the model's clothing to obtain corresponding camera parameters; S14: The front view, back view, left view and right view of the above model clothing and their camera parameters are combined into a clothing image pair, and the front view is specified as the input conditional image, and the other views are used as the real label images that need to be predicted, and then divided into a training set, a validation set and a test set according to 8:1:

1.

3. The method for generating clothing from multiple perspectives according to claim 1, characterized in that: The method of the adaptive conditional feature selection module described in step S3 is: By adaptively selecting the features of clothing views as conditional features to adapt to the specific needs of clothing under different viewing angles, the adaptive conditional feature selection module includes a hard conditional feature selection module, a soft conditional feature selection module and a multi-scale feature extraction module, which respectively realize the extraction of global clothing features, local clothing features and multi-scale clothing features. By utilizing the above modules, the adaptive conditional feature selection module can adaptively and dynamically adjust the feature selection strategy according to the similarity between the input clothing model image and the corresponding clothing posture image.

4. The method for generating clothing from multiple perspectives according to claim 1, characterized in that: The method of the joint attention-based diffusion UNet network described in step S3 is: The diffusion UNet network based on joint attention adopts a downsampling-upsampling framework, in which the image resolution is gradually reduced in downsampling to extract features, and the image resolution is gradually restored in upsampling to generate multi-view clothing images. The self-attention method and the cross-attention method in the upsampling process are replaced by the cross-view interactive self-attention module and the cascaded cross-attention module. The consistency and quality in the multi-view images are effectively enhanced by using the form of joint attention, and the lost and erroneous texture details are refined through selective fusion, and the ability to retain high-frequency details in the original clothing is enhanced. At the same time, the iterative denoising ability of the diffusion model is retained, and the Gaussian distribution is converted into the target distribution, realizing the generation of high-quality multi-view clothing images from random Gaussian noise.

5. The method for generating clothing from multiple perspectives according to claim 1, characterized in that: The joint loss function described in step S3 is: It includes diffusion loss function, reconstruction loss function and perceptual loss function to ensure the high quality of the generated image at the pixel level and perceptual level. LDM It is used to minimize the difference between the predicted noise and the actual noise to improve the quality of the generated image. The reconstruction loss function L1 calculates the difference between the model-generated image and the original input image through the first normal form as an indicator of image consistency. The perceptual loss function L perc The quality of the generated image in visual perception is usually measured by comparing the difference between the model-generated image and the original image at the feature level. The pre-trained VGG network is selected to extract deep features and calculate the difference between feature vectors, and the weights λ1 and λ perc The optimal balance point of the model is chosen to achieve the best image generation effect; during the model training process, the parameters of the ALC-MVD model are optimized by minimizing the joint loss function to ensure that when the model generates multi-view clothing images, it is not only consistent with the original image at the pixel level, but also close to the real image in visual perception.

6. The method for generating clothing from multiple perspectives according to claim 1, characterized in that: The hard condition feature selection module in step S4 is used to extract global features from the clothing image to support the multi-view generation process, ensure that the global features of the clothing remain consistent in each view generated by the multi-view, and improve the quality and realism of the generated image. The specific method is: First, the pre-trained SAM model is used to segment the model clothing image to obtain more accurate clothing areas; Then, the hard conditional feature selection module evaluates the similarity between the clothing pose and the model’s pose, which helps determine which view of the clothing image is most suitable as a representative of the global feature; Subsequently, based on the evaluation results of pose similarity, the module will perform hard selection, that is, select the clothing image closest to the target model's pose to ensure that the extracted clothing global features are highly consistent with the target view; Next, a CLIP-based image encoder is used to extract global features of clothing images to ensure that important semantic information of clothing can be captured; Finally, the segmented clothing images are input into the image encoder for encoding of global features and are input into the UNet model of JA-UDM as hard conditions to guide image reconstruction in the multi-view generation process.

7. The method for generating clothing from multiple perspectives according to claim 1, characterized in that: The soft condition feature selection module in step S4 is used to extract and fuse the key components of local features, and enhance the detail expression and realism of multi-view clothing image generation through careful feature selection and fusion strategy. The specific method is as follows: First, the pre-trained SAM model is used to segment the clothing of the model clothing image in the front view to obtain a more accurate clothing area; Then, a trainable multi-scale feature encoder E is designed l (), extracting multi-scale local features from clothing images in the front view, which contain detailed information such as clothing texture and pattern; Subsequently, the skeleton images of the corresponding model clothing images are converted into posture features through a trainable posture encoder. These skeleton images provide key information about the character's posture. Next, the extracted posture features are combined with multi-scale clothing local features, and the weight adjustment mechanism in the soft conditional feature selection module is used to selectively enhance or weaken the features of different views; Finally, according to the pose similarity and the geometric relationship between views, the fused local features are mapped to the corresponding UNet network scale in JA-UDM as soft condition input to assist the generation of multi-view images. The soft conditional feature selection module adopts a soft selection mechanism to weight the clothing features of different views to meet the needs of multi-view reconstruction. This weighting strategy provides rich local feature support for the ALC-MVD model, allowing the model to flexibly adjust the importance of features in different situations, making the generated multi-view clothing images richer and more realistic in details.

8. The method for generating clothing from multiple perspectives according to claim 1, characterized in that: The cross-view interactive self-attention module in step S4 promotes information interaction and fusion between different views by establishing associations between multi-view image features, thereby achieving multi-dimensional feature representation and detail enhancement. The specific method is as follows: Firstly, a position encoding technique inspired by light field network (LFN) is used to associate a ray to each pixel in the CVI-SA module and represent it using Plücker coordinates. Strict geometric constraints and camera parameters are introduced to achieve accurate pixel-level feature encoding. Then, an adjacent view cross-attention module is designed to aggregate light features from the adjacent views of the target view, realize cross-view interaction through position encoding and projection features, and capture the feature correlation between different views; Next, a novel ray self-attention module is used to simulate the contribution of spatial points to pixel-level features. The depth weight of the sampled rays is modeled through the self-attention mechanism to achieve ray aggregation, revealing the depth relationship and spatial correlation between the sampled points along the rays, and integrating the spatial point features into the pixel features. Finally, the weighted summation method is used to fuse the geometric spatial features into the pixel features of the target unit block to achieve efficient multi-view consistency modeling. In step S4, the cascade cross attention module deeply integrates the global features with the local features by means of attention cascade, so as to further enhance the details and consistency of the multi-view clothing image generation. The specific method is as follows: First, the C-CA module performs self-attention calculation on the current feature to capture the dependencies within the local area and enhance the expressiveness of the local features. Then, based on self-attention, a double cross attention mechanism is introduced, the features currently processed in the module are used as queries, global features and local features are used as keys and values ​​respectively, and a learnable fusion vector λ is designed. c , selectively fuses global and local features in the channel dimension, strengthening the retention of high-frequency details while retaining global semantic information; Next, the global features are refined and error corrected using local features, so that the C-CA module can enhance the details in clothing images, such as textures and patterns, and improve the realism of the generated images. Finally, by operating at different scales, we ensure fine alignment and deep fusion of features at all scales.

9. The method for generating clothing from multiple perspectives according to claim 1, characterized in that: The method for performing reasoning using the adaptive local conditional multi-view diffusion model according to the model clothing image input by the user in step S6 is: S61: Segment the model clothing image input by the user through the pre-trained SAM model to obtain a clothing posture image; S62: Input the model clothing image into the image encoder to encode the latent space features, and input it into the diffusion UNet network based on joint attention; S63: Calculate the camera parameters corresponding to the clothing posture graph and input them into the adaptive condition feature selection module; S64: The global and multi-scale local conditional features output by the adaptive conditional feature selection module are input into the diffusion UNet network based on joint attention, and the alignment and fusion of the current features and the conditional features are achieved through the cross-view interactive self-attention module and the cascaded cross-attention module; S65: Using the fused features, the joint attention-based diffusion UNet network starts from pure noise and gradually denoises it to restore a clear image from the random state to generate multi-view clothing images; S66: The system outputs multi-view clothing images generated by the adaptive local conditional multi-view diffusion model to meet the user's needs for clothing display from different perspectives.

10. A clothing multi-view generation device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, the steps of the method for generating clothing multi-views of the above-mentioned adaptive local condition multi-view diffusion model are implemented.

Images