A method, medium and system for real-time generation of immersive scene fitting videos

Abstract

The present invention provides a real-time generation method, medium and system for immersive scene fitting videos, belonging to the field of immersive scene fitting video technology, including: real-time acquisition of multi-angle images of fitting personnel, and pre-processing; extraction of facial 3D features and clothing 3D features of fitting personnel; combination of pre-trained clothing-body models to obtain the body 3D features of fitting personnel; selection of fitting scene videos from a preset fitting scene video library, and adjustment of the model according to the facial and body features of the fitting personnel; establishment of a personalized model for the fitting personnel and clothing by using the Lora fine-tuning method, used to generate a 3D image of the fitting personnel wearing clothing; fusion of the adjusted facial, body and clothing 3D images into the key frames of the selected video to generate an immersive fitting video; and use of a parallel gray wolf hunting algorithm for fast Lora fine-tuning optimization. Personalized optimization for consumers and clothing is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of immersive scene fitting videos, and in particular, relates to a method, medium and system for real-time generation of immersive scene fitting videos. Background Art

[0002] At present, the rapid development of Internet technology and e-commerce has provided consumers with a brand new shopping experience. In the clothing industry, online fitting has become an indispensable part of the shopping process. Compared with traditional offline shopping malls, consumers can not only greatly improve their shopping efficiency through online fitting, but also better understand the effect of clothing on themselves, so as to make more informed purchasing decisions.

[0003] There are two main ways to implement the existing online fitting technology: one is virtual fitting based on two-dimensional images, and the other is immersive fitting based on three-dimensional models. The former usually uses photos uploaded by consumers to superimpose clothing effects on the images, allowing consumers to intuitively feel the wearing effect of the clothing. This method is relatively simple, but it is difficult to give consumers a real fitting experience because it ignores the human body shape and dynamic effects. The latter uses three-dimensional scanning technology to obtain the consumer's three-dimensional human body model, and simulates the wearing effect of clothing on this basis, so that consumers are placed in an immersive virtual fitting scene. This method can provide a more realistic fitting experience, but it also has some shortcomings: first, the cost of three-dimensional scanning equipment is high, which is difficult for ordinary consumers to accept; second, even if the consumer's three-dimensional human body model is obtained, it is difficult to restore the consumer's real appearance and clothing wearing effect in the virtual scene.

[0004] In response to the above problems, the industry has proposed an immersive virtual fitting technology based on computer vision and deep learning. This method does not require special 3D scanning equipment, but uses 2D photos uploaded by consumers to construct a 3D human model of the consumer through computer vision technologies such as 3D reconstruction and human feature extraction. At the same time, using the deep learning model, the 3D effect of the consumer wearing the clothing can be accurately predicted based on the 2D image and 3D features of the clothing. Finally, the adjusted 3D human model is integrated into the pre-designed virtual fitting scene to form an immersive virtual fitting experience. This method can not only provide a more realistic fitting effect, but also be more convenient and easy to use for consumers. Since the existing deep learning models are usually trained for general clothing types and body types, it is difficult to perform personalized optimization for different consumers and clothing, making it difficult to achieve the ideal fitting effect. Summary of the invention

[0005] In view of this, the present invention provides a method, medium and system for real-time generation of immersive scene fitting videos, which can solve the technical problem that existing deep learning models are usually trained for general clothing types and body types, and it is difficult to perform personalized optimization for different consumers and clothing, thus making it difficult to achieve ideal fitting effects.

[0006] The present invention is achieved in that:

[0007] A first aspect of the present invention provides a method for real-time generation of an immersive scene fitting video, comprising the following steps:

[0008] S10, acquiring multiple images of the fitting personnel wearing the fitting clothes acquired by multiple cameras at different shooting angles in real time, and pre-processing to obtain multiple basic images of the fitting personnel;

[0009] S20, for the multiple basic images, extracting facial features and clothing image features of the person trying on clothes in each of the basic images, and obtaining three-dimensional facial features and three-dimensional clothing features of the person trying on clothes by three-dimensional reconstruction according to the shooting angle;

[0010] S30, inputting the three-dimensional features of the clothing into a pre-trained clothing-body model to obtain the three-dimensional features of the body of the person fitting the clothes;

[0011] S40, acquiring a fitting scene video selected by a fitting person as a selected video from a pre-set fitting scene video library, wherein the fitting scene video library includes a plurality of fitting scene videos, each fitting scene video being an action video of a model wearing clothes in a corresponding scene;

[0012] S50, adjusting the facial features and contours of the model in the key frame of the selected video based on the three-dimensional facial features and three-dimensional body features of the fitting person;

[0013] S60, using a plurality of basic images, performing Lora fine-tuning on the clothing-body model to obtain a personalized Lora model for the fitting person and the fitting clothing, which is used to accurately infer a three-dimensional image of the fitting person wearing the fitting clothing from the three-dimensional features of the clothing and the clothing image;

[0014] S70, in each key frame, according to the model's body image, the model's body three-dimensional features are calculated and input into the clothing-body model+Lora model, so as to obtain an adjusted clothing three-dimensional image of the model wearing the try-on clothing;

[0015] S80, fusing the adjusted facial features, body features and three-dimensional image of the model into the corresponding key frames of the selected video to form a key frame sequence of the fitting video, and finally generating an immersive video of the fitting person trying on clothes in the selected scene by interpolation method.

[0016] The Lora fine-tuning step uses a parallel gray wolf hunting algorithm for rapid optimization, and the specific steps include:

[0017] S61, loading the input multiple basic images into the GPU video memory;

[0018] S62, using CUDA parallel computing, starting multiple threads to respectively perform Lora fine-tuning tasks, wherein each thread is responsible for fine-tuning parameters of the clothing-body model corresponding to a basic image;

[0019] S63, each thread uses a parallel gray wolf hunting algorithm to perform rapid optimization, specifically, initialize the gray wolf population, each gray wolf represents a set of Lora model parameters, parallelly calculate the loss function value corresponding to each gray wolf, update the positions of the alpha wolf, beta wolf and delta wolf according to the loss function value, and update the positions of other gray wolves in parallel according to the positions of the alpha wolf, beta wolf and delta wolf, and repeat the above steps until the maximum number of iterations or convergence conditions are reached;

[0020] S64, stitching the Lora model parameters optimized by each thread to form a personalized Lora model for the fitting person and the fitting clothes.

[0021] Among them, the Lora fine-tuning algorithm uses a parallel gray wolf hunting algorithm, which simulates the social behavior of gray wolves during hunting. It calculates the loss function value corresponding to each gray wolf in parallel, and updates the positions of alpha wolves, beta wolves and delta wolves according to the loss function value, and then updates the positions of other wolves, so as to quickly find the optimal Lora model parameters. Generally, the maximum number of iterations is set to 1000; or the convergence condition is set to be as large as the maximum number of iterations, or the loss function value changes less than 1e-5 for 10 consecutive iterations.

[0022] Furthermore, the preprocessing method includes image correction, human body detection, human body segmentation, and facial feature point detection.

[0023] The step S10 specifically comprises the following steps:

[0024] Step 101, using multiple cameras to capture multi-angle images of the person trying on clothes in real time when the person is wearing the clothes being tried on;

[0025] Step 102: pre-process the multi-angle images, including image correction, human body detection, human body segmentation, facial feature point detection, etc., to obtain a plurality of basic images of the people being fitted for clothing.

[0026] The method for extracting facial features of the person fitting clothes in each of the basic images is to adopt the OpenPose method.

[0027] The step S20 specifically comprises the following steps:

[0028] Step 201: extract facial features of the person fitting clothes in each basic image, including the position and shape information of key feature points such as eyes, nose, lips, etc.;

[0029] Step 202: extracting image features of the tried-on clothing in each basic image, including information such as texture, color, and shape of the clothing;

[0030] Step 203: Reconstruct the three-dimensional facial model and three-dimensional clothing model of the person being fitted for clothing according to the geometric relationship between images from different perspectives by using structured light scanning or binocular vision.

[0031] The step S30 specifically comprises the following steps:

[0032] Step 301, inputting the clothing three-dimensional features obtained in step S20 into a pre-trained clothing-body deep neural network model;

[0033] Step 302: The clothing-body model is trained using a large number of clothing samples and corresponding three-dimensional human body models, and can predict the three-dimensional body features of the person being fitted based on the input three-dimensional features of the clothing.

[0034] In step S40, each fitting scene has action videos of models of different genders, ages, and body types, and each fitting scene video has key frames marked.

[0035] The step S50 specifically includes the following steps:

[0036] Step 501: Mapping the three-dimensional facial features of the fitting person to the facial features and contours of the model in the selected video, so that the facial features and contours of the model are more similar to those of the fitting person;

[0037] Step 502: According to the three-dimensional features of the person fitting the clothes, the body features of the model in the selected video are adjusted to make the body shape of the model closer to that of the person fitting the clothes.

[0038] The clothing-body model training step specifically includes:

[0039] Establishing a training data set, including multiple sets of clothing-body data, wherein the clothing-body data are three-dimensional body features of multiple models wearing the tried-on clothing and their corresponding three-dimensional clothing features of the tried-on clothing;

[0040] A training model is provided, wherein a bidirectional recurrent neural network is trained using the training data set to obtain a clothing-body model.

[0041] The bidirectional recurrent neural network uses a bidirectional BP neural network.

[0042] The step S70 specifically includes:

[0043] Step 701, analyzing the action and posture information of the model in each key frame in the selected video, and calculating the three-dimensional features of the model's body;

[0044] Step 702: input the three-dimensional features of the body into the personalized clothing-body Lora model trained in step S60;

[0045] Step 703: using the Lora model, output a three-dimensional clothing image of the model wearing the clothing according to the input body features and clothing features.

[0046] The step S80 specifically includes:

[0047] Step 801, overlaying the adjusted model facial features obtained in step S50 onto the model face of the corresponding key frame in the selected video;

[0048] Step 802: applying the adjusted model body features obtained in step S50 to the model body of the corresponding key frame in the selected video;

[0049] Step 803, merging the clothing 3D image obtained in step S70 onto the model clothing of the corresponding key frame in the selected video;

[0050] Step 804: Convert the fused key frame sequence into a continuous immersive fitting video by interpolation or other methods.

[0051] Furthermore, the loss function is set as the difference between the three-dimensional features of the clothing and the three-dimensional features of the body after alignment.

[0052] Finally, an immersive video of a person trying on clothes in a selected scene is generated by an interpolation method, wherein the interpolation method adopts linear interpolation or spline interpolation.

[0053] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned method for real-time generation of an immersive scene fitting video.

[0054] A third aspect of the present invention provides a real-time generation system for immersive scene fitting videos, which includes the above-mentioned computer-readable storage medium.

[0055] Compared with the prior art, the method, medium and system for real-time generation of immersive scene fitting videos provided by the present invention have the following beneficial effects:

[0056] 1. Multi-view image input to achieve accurate 3D human body model construction. Compared with the existing technology, the present invention uses multiple cameras to capture images of consumers from different angles, and uses 3D reconstruction technology to reconstruct a detailed 3D human body model of the consumer, including facial features and clothing features. This method can more accurately restore the actual appearance of the consumer and the effect of wearing clothing, thereby greatly improving the accuracy and realism of the fitting results.

[0057] 2. Based on the deep learning optimization of the personalized Lora model, targeted fitting effect prediction is achieved. The present invention adopts the Lora fine-tuning technology, and further uses the actual image data of consumers to optimize the model on the basis of the general clothing-body prediction model. This can better capture the personal characteristics of consumers, thereby predicting more accurate clothing wearing effects.

[0058] 3. Preset immersive fitting scenes provide an immersive virtual fitting experience. Compared with a single pure virtual scene, the present invention uses a variety of pre-designed fitting scene video materials, including different environments, different models, etc., which can allow consumers to experience the wearing effect of clothing more immersively. This immersive fitting scene greatly enhances consumers' fitting experience, thereby improving the overall shopping experience.

[0059] In general, the immersive virtual fitting technology proposed in the present invention makes full use of cutting-edge technologies such as computer vision and deep learning, solves key problems existing in existing solutions, and provides consumers with a more accurate, personalized, and immersive fitting experience. It solves the technical problem that existing deep learning models are usually trained for general clothing types and body types, and it is difficult to perform personalized optimization for different consumers and clothing, thus it is difficult to achieve ideal fitting effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0061] Figure 1A flow chart of the method provided by the present invention;

[0062] Figure 2 is a detailed flow chart of step S60; DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0064] like Figure 1 FIG. 1 is a flow chart of a method for real-time generation of immersive scene fitting videos provided by the present invention. The method comprises the following steps:

[0065] S10, acquiring multiple images of the fitting personnel wearing the fitting clothes acquired by multiple cameras at different shooting angles in real time, and pre-processing to obtain multiple basic images of the fitting personnel;

[0066] S20, for the multiple basic images, extracting facial features and clothing image features of the person trying on clothes in each of the basic images, and obtaining three-dimensional facial features and three-dimensional clothing features of the person trying on clothes by three-dimensional reconstruction according to the shooting angle;

[0067] S30, inputting the three-dimensional features of the clothing into a pre-trained clothing-body model to obtain the three-dimensional features of the body of the person fitting the clothes;

[0068] S40, obtaining a fitting scene video selected by the fitting person as a selected video from a pre-set fitting scene video library, wherein the fitting scene video library includes a plurality of fitting scene videos, each fitting scene video is an action video of a model wearing clothes in a corresponding scene, wherein each fitting scene has action videos of models of different genders, ages, and body types, and each fitting scene video has key frames marked;

[0069] S50, adjusting the facial features and contours of the model in the key frame of the selected video based on the three-dimensional facial features and three-dimensional body features of the fitting person;

[0070] S60, using a plurality of basic images, performing Lora fine-tuning on the clothing-body model to obtain a personalized Lora model for the fitting person and the fitting clothing, which is used to accurately infer a three-dimensional image of the fitting person wearing the fitting clothing from the three-dimensional features of the clothing and the clothing image;

[0071] S70, in each key frame, according to the model's body image, the model's body three-dimensional features are calculated and input into the clothing-body model+Lora model, so as to obtain an adjusted clothing three-dimensional image of the model wearing the try-on clothing;

[0072] S80, the adjusted facial features, body features and clothing 3D images of the model are fused into the corresponding key frames of the selected video to form a key frame sequence of the fitting video. Finally, an immersive video of the fitting person trying on clothes in the selected scene is generated by interpolation and other methods.

[0073] The Lora fine-tuning step uses a parallel gray wolf hunting algorithm for rapid optimization, and the specific steps include:

[0074] S61, loading the input multiple basic images into the GPU video memory;

[0075] S62, using CUDA parallel computing, starting multiple threads to respectively perform Lora fine-tuning tasks, wherein each thread is responsible for fine-tuning parameters of the clothing-body model corresponding to a basic image;

[0076] S63, each thread uses a parallel gray wolf hunting algorithm to perform rapid optimization, specifically, initialize the gray wolf population, each gray wolf represents a set of Lora model parameters, parallelly calculate the loss function value corresponding to each gray wolf, update the positions of the alpha wolf, beta wolf and delta wolf according to the loss function value, and update the positions of other gray wolves in parallel according to the positions of the alpha wolf, beta wolf and delta wolf, and repeat the above steps until the maximum number of iterations or convergence conditions are reached;

[0077] S64, stitching the Lora model parameters optimized by each thread to form a personalized Lora model for the fitting person and the fitting clothes.

[0078] The specific implementation methods of the above steps are described in detail below:

[0079] The specific implementation method of step S10 is as follows: first, the system acquires multiple images of the fitting personnel wearing the fitting clothes acquired by multiple cameras with different shooting angles in real time. These images contain the facial features and clothing features of the fitting personnel. In order to further extract and utilize these features, the images need to be preprocessed. The main steps of preprocessing include: image correction, human body detection, human body segmentation, facial feature point detection, etc. After preprocessing, multiple basic images of the fitting personnel can be obtained.

[0080] The specific implementation of step S20 is: for the multiple basic images obtained in step S10, first extract the facial features and clothing image features of the person trying on clothes in each basic image. Facial features include the position and shape information of key feature points such as eyes, nose, lips, etc. Clothing image features include information such as texture, color, shape, etc. of clothing.

[0081] In order to make full use of the information obtained from multiple perspectives, a 3D reconstruction method is used to estimate the 3D facial features of the person trying on clothes and the 3D features of the clothes. Specifically, by using structured light scanning or binocular vision methods, the 3D facial model and 3D clothing model of the person trying on clothes are reconstructed by analyzing the geometric relationship between images from different perspectives. This not only obtains richer feature information, but also better describes the posture of the person trying on clothes in space and the wearing condition of the clothes.

[0082] The specific implementation method of step S30 is: input the three-dimensional features of the clothing obtained in step S20 into a pre-trained clothing-body model. This clothing-body model is a deep neural network model, and its training data includes a large number of clothing samples and corresponding human body three-dimensional models. Through this model, the three-dimensional features of the body of the person fitting the clothes can be predicted based on the input three-dimensional features of the clothing. In other words, the three-dimensional model of the human body wearing the clothing can be inferred based on the size, shape and other information of the clothing.

[0083] The specific implementation of step S40 is as follows: first, the system pre-establishes a fitting scene video library, which contains multiple fitting scene videos. Each fitting scene video is shot in a corresponding scene, and the models in the video wear different types of clothing and perform actions in the scene. The models in these videos have different genders, ages and body shapes. Each fitting scene video has key frames marked in advance.

[0084] When the user selects the scene in which they want to try on clothes, the system will extract the corresponding video from the fitting scene video library as the selected video. This selected video will serve as the basic material for the subsequent synthesis of fitting videos.

[0085] The specific implementation of step S50 is as follows: first, the system adjusts the facial features and contours of the model in the selected video based on the three-dimensional facial features of the fitting person obtained in step S20. Specifically, the three-dimensional facial features of the fitting person are mapped to the face of the model in the selected video, so that the facial features and contours of the model are more similar to those of the fitting person. This can enhance the realism and immersion of the fitting video.

[0086] At the same time, the body features of the model in the selected video will be adjusted based on the three-dimensional body features of the fitting personnel obtained in step S30. The body features of the fitting personnel will be applied to the model, so that the model's body shape is closer to the fitting personnel.

[0087] Through the above-mentioned adjustment of facial and body features, the model in the selected video can be made closer to the actual appearance of the fitting person, thereby enhancing the realism of the fitting video.

[0088] The specific implementation of step S60 is: based on the multiple basic images obtained in step S10, a personalized Lora model for the current fitting person and the clothing being fitted is trained using the Lora fine-tuning method. Lora is a fast fine-tuning technology based on a pre-trained model, which can quickly obtain a personalized model under limited training data conditions.

[0089] Specifically, we first load multiple basic images into the GPU memory. Then, we use the parallel computing capability of CUDA to start multiple threads to perform Lora fine-tuning tasks, with each thread responsible for fine-tuning a basic image. In each thread, we use the parallel gray wolf hunting algorithm to quickly find the optimal solution.

[0090] The gray wolf hunting algorithm is a swarm intelligence algorithm that simulates the social behavior of gray wolves during hunting. The algorithm initializes a group of gray wolves, each of which represents a set of Lora model parameters. The algorithm calculates the loss function value corresponding to each gray wolf in parallel, and then updates the positions of the alpha wolf, beta wolf, and delta wolf according to the loss function value, and updates the positions of other wolves according to the positions of these three wolves. By iterating this process, a set of optimal Lora model parameters can be quickly found, that is, a personalized Lora model for the current person trying on clothes and the clothes being tried on.

[0091] The Lora model parameters optimized by each thread are combined to form the personalized Lora model used in the end. This Lora model can be used to accurately infer the three-dimensional image of the person wearing the clothing from the three-dimensional features of the clothing and the clothing image.

[0092] The specific implementation of step S70 is as follows: first, the three-dimensional features of the model's body in each key frame in the selected video need to be calculated. This can be achieved by analyzing the model's movements and posture information in the video.

[0093] Then, the calculated three-dimensional features of the model's body are input into the clothing-body model and the personalized Lora model trained in step S60. The two models can output a three-dimensional clothing image of the model wearing the tried-on clothing according to the input body features and clothing features.

[0094] Through this step, a 3D clothing image of the person trying on the clothing can be generated in each key frame. These 3D clothing images will provide basic materials for subsequent video synthesis.

[0095] The specific implementation method of step S80 is: finally, the adjusted facial features, body features and three-dimensional image of the model obtained in step S50 and step S70 need to be merged into the corresponding key frame of the selected video. Specifically, the facial features of the fitting personnel need to be overlaid on the face of the model, the body features of the fitting personnel need to be applied to the body of the model, and the three-dimensional image of the clothing needs to be merged onto the clothing of the model.

[0096] Through this series of fusion operations, the key frame sequence of the fitting video is obtained. Finally, these key frame sequences can be converted into a continuous and immersive fitting video by using interpolation and other methods.

[0097] In order to better understand the method of the present invention, the specific implementation of the method is described in more detail below in conjunction with specific formulas:

[0098] The specific implementation method of step S10:

[0099] Multiple images of people wearing the tried-on clothes are acquired in real time from multiple cameras with different shooting angles, and multiple basic images of people being tried on are obtained through preprocessing.

[0100] Specifically, there are n cameras that capture images of people trying on clothes at different angles. For the i-th camera, the image it captures is recorded as Where H and W represent the height and width of the image respectively, and 3 represents the three RGB channels.

[0101] In order to further extract and utilize these image features, the acquired original images need to be preprocessed. The main steps of preprocessing include:

[0102] 1. Image correction: Due to the different installation positions and angles of the camera, the acquired image may have problems such as distortion and rotation, and corresponding correction is required to make the image conform to the standard coordinate system. The corrected image is recorded as

[0103] 2. Human detection: Use mature target detection algorithms, such as Faster R-CNN, to detect human bodies in the rectified images. The position of the person trying on clothes is located in the image and the human body bounding box is obtained.

[0104] 3. Human body segmentation: Use semantic segmentation network, such as DeepLabV3+, to segment the corrected image Perform pixel-level human segmentation to obtain a fine segmentation mask M of the fitting personnel i .

[0105] 4. Facial feature point detection: Use key point detection algorithms, such as OpenPose, to detect facial features in the human body segmentation results Mi The key points of the tester's face are inspected, including the positions of eyes, nose, lips and other feature points.

[0106] After the above preprocessing steps, we get a number of basic images of people trying on clothes, denoted as Where P i represents the set of facial feature points of the person trying on clothes in the i-th image. These basic images will be used as input for subsequent steps.

[0107] The specific implementation method of step S20:

[0108] For the multiple basic images obtained in step S10, the facial features and clothing image features of the person trying on clothes in each basic image are extracted, and three-dimensional facial features and clothing three-dimensional features of the person trying on clothes are obtained by three-dimensional reconstruction according to the shooting angle.

[0109] 1. Facial feature extraction:

[0110] For the i-th basic image First, use the detected facial key points P i , extract the facial geometric features of the fitting personnel, including the position and shape information of key points such as eyes, nose, lips, etc. These features can be represented as a facial feature vector

[0111] 2. Clothing feature extraction:

[0112] In the human body segmentation mask M i Under the guidance of The texture, color, shape and other visual features of the tried-on clothing are extracted from the image. These features can be represented as a clothing feature vector

[0113] 3. 3D reconstruction:

[0114] In order to make full use of the information obtained from multiple perspectives, a 3D reconstruction method is used to estimate the 3D facial features of the person trying on clothes and the 3D features of the clothes. Specifically, by using structured light scanning or binocular vision, images from different perspectives are analyzed. The geometric relationship between the two images is used to reconstruct the three-dimensional facial model of the fitting personnel. and 3D clothing models Where N and M represent the number of vertices of the face and clothing models, respectively.

[0115] In summary, through step S20, the facial feature vector of the fitting person is obtained. Clothing feature vector And the corresponding 3D facial model M faceand 3D clothing model M cloth . These features will provide important input for subsequent steps.

[0116] The specific implementation method of step S30:

[0117] The clothing three-dimensional feature M obtained from step S20 cloth Input into the pre-trained clothing-body model to obtain the three-dimensional body features of the person trying on clothes.

[0118] Specifically, a deep neural network model is used as the clothing-body model, denoted as The training data of this model includes a large number of clothing samples and corresponding 3D human body models.

[0119] Given the input clothing 3D feature M cloth , can be achieved through clothing-body model Predict the three-dimensional features of the person trying on clothes Where K represents the number of vertices of the shape model.

[0120] The mathematical expression is as follows:

[0121]

[0122] In this way, the 3D features of the person being fitted can be inferred based on the input 3D features of the clothing, which provides important input for subsequent video synthesis.

[0123] Specific implementation of step S40:

[0124] From the pre-set fitting scene video library, a fitting scene video selected by the fitting personnel is obtained as the selected video. The fitting scene video library includes multiple fitting scene videos, each fitting scene video is an action video of a model wearing clothes in a corresponding scene, wherein each fitting scene has action videos of models of different genders, ages, and body shapes, and each fitting scene video has key frames marked.

[0125] Assume that there are M fitting scene videos in the fitting scene video library, recorded as Each fitting scene video V m All contain the model action sequence in the corresponding scene, which can be expressed as:

[0126]

[0127] in represents the tth frame image of the mth fitting scene video, T m Indicates the total number of frames in the video.

[0128] In addition, each fitting scene video V mThe keyframe sets are already marked in advance

[0129] When the user selects the scene in which he wants to try on clothes, the system will extract the corresponding video V from the fitting scene video library. m As the selected video, this selected video will serve as the basic material for the subsequent synthetic fitting video.

[0130] The specific implementation method of step S50:

[0131] Based on the three-dimensional facial features of the fitting personnel face and the three-dimensional features of the shape M body , adjusting the facial features and contours of the model in the key frames of the selected video.

[0132] Specifically, for the selected video V m Each key frame need:

[0133] 1. The three-dimensional facial features of the fitting personnel M face Mapped to the model's facial features, making the model's facial features and contours more similar to those of the person trying on the clothes. This can be done through a facial feature transfer function To achieve:

[0134]

[0135] in Represents the adjusted keyframe image.

[0136] 2. At the same time, it will also be based on the three-dimensional features of the fitting personnel M body , adjust the model's body features to make the model's body shape closer to the person trying on the clothes. This can be done through a body feature transfer function To achieve:

[0137]

[0138] Through the above-mentioned adjustment of facial and body features, the model in the selected video can be made closer to the actual appearance of the fitting person, thereby enhancing the realism of the fitting video.

[0139] Specific implementation of step S60:

[0140] Based on the multiple basic images obtained in step S10, the Lora fine-tuning method is used to train a personalized Lora model for the current person trying on clothes and the clothes being tried on.

[0141] Suppose there are n basic images First, these images are loaded into the GPU memory. Then, using the CUDA parallel computing capability, K threads are started to perform Lora fine-tuning tasks respectively, and each thread is responsible for the fine-tuning of a basic image.

[0142] In each thread, a parallel gray wolf hunting algorithm is used for fast optimization. The algorithm initializes a group of L gray wolves, each gray wolf represents a set of Lora model parameters The algorithm calculates the loss function value corresponding to each gray wolf in parallel Then the positions of alpha wolf, beta wolf and delta wolf are updated according to the loss function value, and the positions of other wolves are updated according to the positions of these three wolves.

[0143] The mathematical description is as follows:

[0144] 1. Initialize the gray wolf group:

[0145]

[0146] Where d represents the parameter dimension of the Lora model.

[0147] 2. The loss function is set as the difference between the clothing 3D features and the body 3D features after alignment, and the loss function value corresponding to each gray wolf is calculated:

[0148]

[0149] 3. Update the positions of alpha wolf, beta wolf and delta wolf:

[0150]

[0151] where w α ,w β ,w δ Represents the positions of alpha wolf, beta wolf and delta wolf respectively, is the random coefficient, is a random number, and α is the step size parameter.

[0152] 4. Update the location of other gray wolves:

[0153]

[0154] in is the random coefficient, is a random number.

[0155] 5. Repeat steps 2-4 until the maximum number of iterations T or convergence condition is reached

[0156] By iterating the above steps, we can quickly find a set of optimal Lora model parameters. That is, a personalized Lora model for the current fitting person and the clothing being tried on.

[0157] The Lora model parameters optimized by each thread are spliced ​​together to form the personalized Lora model used in the end. where w * is the optimal parameter vector after splicing. This Lora model can be used to extract the clothing 3D features M cloth and clothing images to accurately infer the three-dimensional image of the person wearing the clothing.

[0158] Specific implementation of step S70:

[0159] In each key frame, the three-dimensional features of the model's body are calculated based on the model's body image and input into the clothing-body model+Lora model to obtain an adjusted three-dimensional image of the model wearing the try-on clothing.

[0160] Specifically, for the selected video V m Each key frame need:

[0161] 1. Analyze the model’s movements and posture information in the key frame, and calculate the model’s three-dimensional body features through the human body posture estimation algorithm

[0162] 2. The three-dimensional features of the shape Input to the personalized clothing-body Lora model trained in step S60 In this paper, the Lora model is used to input the body features and clothing features M cloth , output the 3D clothing image of the model wearing the clothing after trying it on

[0163] The mathematical expression is as follows:

[0164]

[0165] Through this step, a 3D clothing image of the person trying on the clothing can be generated in each key frame. These 3D clothing images will provide basic materials for subsequent video synthesis.

[0166] The specific implementation method of step S80:

[0167] The adjusted facial features, body features and three-dimensional images of the model are fused into the corresponding key frames of the selected video to form a key frame sequence of the fitting video. Finally, an immersive video of the fitter trying on clothes in the selected scene is generated through interpolation and other methods.

[0168] Specifically:

[0169] 1. The adjusted model facial features obtained in step S50 Overlay to selected video V m Corresponding keyframes Model's face.

[0170] 2. Apply the adjusted model body features obtained in step S50 to the model body of the corresponding key frame in the selected video.

[0171] 3. The clothing 3D image obtained in step S70 Fusion to the model clothing of the corresponding key frame in the selected video. The fusion method can be texture mapping, 3D model embedding, etc.

[0172] 4. The fused key frame sequence It is converted into a continuous immersive fitting video through video interpolation algorithms such as linear interpolation and spline interpolation.

[0173] The relevant variables are explained as follows: n: the number of cameras; H, W: the height and width of the image; I i : The original image captured by the i-th camera; The i-th image after correction; M i : The segmentation mask of the person trying on clothes in the i-th image; P i : The set of facial key points of the person trying on clothes in the i-th image; The facial feature vector of the person trying on clothes in the i-th image; The clothing feature vector of the person trying on clothes in the i-th image; M face : 3D facial model of the fitting personnel; M cloth : 3D clothing model of the fitting personnel; M body : 3D body model of the fitting personnel; K: number of vertices of the body model; M: number of fitting scene videos; V m :The mth fitting scene video; T m : The total number of frames of the mth fitting scene video; The tth frame image of the mth fitting scene video; The key frame set of the mth fitting scene video; The tth frame image of the mth video after facial and body feature adjustment; L: the number of gray wolves in the gray wolf algorithm; w l: Lora model parameters represented by the first gray wolf; d: parameter dimension of the Lora model; w α ,w β ,w δ : The positions of alpha wolf, beta wolf and delta wolf; c1, c2, c3, c4, c5, c6: random coefficients in the gray wolf algorithm; r1, r2, r3, r4, r5, r6: random numbers in the gray wolf algorithm; α: step size parameter in the gray wolf algorithm; T: maximum number of iterations of the gray wolf algorithm; ∈: convergence condition of the gray wolf algorithm; w * : The final optimized Lora model parameters; The 3D clothing image of the model wearing the clothing in the tth frame of the mth video; d face : The dimension of the facial feature vector; d cloth : the dimension of clothing feature vector; N: the number of vertices of the facial model; d: the parameter dimension of the Lora model; i: represents the i-th camera or image; m: represents the m-th fitting scene video; t: represents the t-th frame of the m-th fitting scene video; l: represents the l-th gray wolf; Clothing-body model; Facial feature transfer function; Shape feature transfer function.

[0174] Through the above steps, we finally get a highly realistic and immersive fitting video that can truly show the effect of the person trying on the clothes in the selected scene.

[0175] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned method for real-time generation of an immersive scene fitting video.

[0176] A third aspect of the present invention provides a real-time generation system for immersive scene fitting videos, which includes the above-mentioned computer-readable storage medium.

[0177] Specifically, the principle of the present invention is: this solution mainly includes three key steps: multi-view image acquisition and 3D reconstruction, deep learning optimization based on personalized Lora model, and integration of preset immersive fitting scenes. These three steps work together to achieve the technical objectives of the present invention.

[0178] First, in the image acquisition phase, the system uses multiple cameras to capture photos of consumers wearing clothing from different angles. These multi-angle image data lay the foundation for subsequent three-dimensional reconstruction. By analyzing the facial feature points, clothing texture and other information in these images, combined with the three-dimensional reconstruction algorithm, a detailed three-dimensional human body model of the consumer can be reconstructed, including the three-dimensional features of the face and clothing. Compared with a single-view two-dimensional image, this multi-view reconstruction method can more accurately restore the consumer's true appearance and clothing effect.

[0179] Secondly, in the deep learning optimization phase, Lora fine-tuning technology is used to train a personalized model for specific consumers and clothing. Lora is a fast fine-tuning method based on a pre-trained model that can quickly obtain a customized model under limited training data conditions. Using multi-threaded parallel computing, the Gray Wolf algorithm is used to quickly optimize the Lora model parameters, thereby obtaining a deep learning model that can accurately predict the effect of the consumer wearing the clothing. This personalized model optimization can greatly improve the accuracy and fit of the fitting results.

[0180] Finally, in the virtual fitting scene integration phase, the adjusted consumer 3D human body model is integrated into the pre-designed immersive fitting scene materials, including different environments and different models. Through fine adjustments to the facial and body features of the models, the models in the virtual scene are closer to the actual image of the consumers. At the same time, the optimized 3D effect of the clothing is superimposed on the model to form the final fitting video. This method of integrating the personality characteristics of consumers into the preset scene greatly enhances the immersion and realism of the fitting experience.

[0181] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for real-time generation of immersive scene fitting videos, characterized in that: The following steps are involved: S10, acquiring multiple images of the fitting personnel wearing the fitting clothes acquired by multiple cameras at different shooting angles in real time, and pre-processing to obtain multiple basic images of the fitting personnel; S20, for the multiple basic images, extracting facial features and clothing image features of the person trying on clothes in each of the basic images, and obtaining three-dimensional facial features and three-dimensional clothing features of the person trying on clothes by three-dimensional reconstruction according to the shooting angle; S30, inputting the three-dimensional features of the clothing into a pre-trained clothing-body model to obtain the three-dimensional features of the body of the person fitting the clothes; S40, acquiring a fitting scene video selected by a fitting person as a selected video from a pre-set fitting scene video library, wherein the fitting scene video library includes a plurality of fitting scene videos, each fitting scene video being an action video of a model wearing clothes in a corresponding scene; S50, adjusting the facial features and contours of the model in the key frame of the selected video based on the three-dimensional facial features and three-dimensional body features of the fitting person; S60, using a plurality of basic images, performing Lora fine-tuning on the clothing-body model to obtain a personalized Lora model for the fitting person and the fitting clothing, which is used to accurately infer the three-dimensional image of the fitting person wearing the fitting clothing from the three-dimensional features of the clothing and the clothing image; S70, in each key frame, according to the model's body image, the model's body three-dimensional features are calculated and input into the clothing-body model+Lora model, so as to obtain an adjusted clothing three-dimensional image of the model wearing the try-on clothing; S80, fusing the adjusted facial features, body features and three-dimensional image of the model into the corresponding key frames of the selected video to form a key frame sequence of the fitting video, and finally generating an immersive video of the fitting person trying on clothes in the selected scene by interpolation method.

2. The method for real-time generation of immersive scene fitting videos according to claim 1, characterized in that: The Lora fine-tuning step uses a parallel gray wolf hunting algorithm for rapid optimization, and the specific steps include: S61, loading the input multiple basic images into the GPU video memory; S62, using CUDA parallel computing, starting multiple threads to respectively perform Lora fine-tuning tasks, wherein each thread is responsible for fine-tuning parameters of the clothing-body model corresponding to a basic image; S63, each thread uses a parallel gray wolf hunting algorithm to perform rapid optimization, specifically, initialize the gray wolf population, each gray wolf represents a set of Lora model parameters, parallelly calculate the loss function value corresponding to each gray wolf, update the positions of the alpha wolf, beta wolf and delta wolf according to the loss function value, and update the positions of other gray wolves in parallel according to the positions of the alpha wolf, beta wolf and delta wolf, and repeat the above steps until the maximum number of iterations or convergence conditions are reached; S64, stitching the Lora model parameters optimized by each thread to form a personalized Lora model for the fitting person and the fitting clothes.

3. The method for real-time generation of immersive scene fitting videos according to claim 2, characterized in that: The preprocessing method includes image correction, human body detection, human body segmentation, and facial feature point detection.

4. The method for real-time generation of immersive scene fitting videos according to claim 1, characterized in that: The method for extracting facial features of the person fitting clothes in each of the basic images is to adopt the OpenPose method.

5. The method for real-time generation of immersive scene fitting videos according to claim 1, characterized in that: In step S40, each fitting scene has action videos of models of different genders, ages, and body types, and each fitting scene video has key frames marked.

6. The method for real-time generation of immersive scene fitting videos according to claim 1, characterized in that: The training steps of the clothing-body model specifically include: Establishing a training data set, including multiple sets of clothing-body data, wherein the clothing-body data are three-dimensional body features of multiple models wearing the tried-on clothing and their corresponding three-dimensional clothing features of the tried-on clothing; A training model is provided, wherein a bidirectional recurrent neural network is trained using the training data set to obtain a clothing-body model.

7. The method for real-time generation of immersive scene fitting videos according to claim 2, characterized in that: The loss function is set as the difference between the clothing three-dimensional features and the body three-dimensional features after alignment.

8. The method for real-time generation of immersive scene fitting videos according to claim 1, characterized in that: Finally, an immersive video of a person trying on clothes in a selected scene is generated by an interpolation method, wherein the interpolation method adopts linear interpolation or spline interpolation.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the method for real-time generation of an immersive scene fitting video according to any one of claims 1 to 8.

10. A real-time generation system for immersive scene fitting videos, characterized in that: A computer-readable storage medium comprising the computer-readable storage medium of claim 9.

Images