Trial-on experience method for 3D digital ethnic costumes

Abstract

The invention discloses a fitting experience method for 3D digital ethnic costumes, and the method comprises the following steps: S1, obtaining the body structure and data information of ethnic costumes, and generating a three-dimensional ethnic costume model; s2, carrying out dynamic simulation, creating a multi-detail-level model, finishing post-processing of the three-dimensional ethnic costume model, and integrating the post-processed three-dimensional ethnic costume model into a database; s3, based on deep learning and data acquisition, completing collection and classification of ethnic costume data, and generating a database; s4, recognizing the position and posture of the user, forming a three-dimensional model of the user, and capturing the body contour and action of the user in real time; s5, the user selects the ethnic costume to be tried on; and S6, rendering the three-dimensional ethnic clothing model to the image of the user in real time, and performing fusion to generate a three-dimensional fitting model. According to the try-on experience method, the figure information between the users is obtained, and the virtual figures corresponding to the users try on the virtual clothes in the virtual scene to be displayed, so that the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual fitting technology, and in particular to a fitting experience method for 3D digital ethnic clothing. Background Art

[0002] In recent years, with the development of network communication technology and image rendering technology, clothing sales can be increasingly carried out with the help of computer software and hardware platforms. Virtual fitting is an emerging clothing shopping model that is widely sought after by consumers.

[0003] Currently, virtual fitting is mainly based on 3D virtual fitting technology. All models and clothes are three-dimensional animations. The advantage is that you can set the model's body proportions to make it more suitable for your own body. The disadvantage is that the cost is too high. A special person has to make the three-dimensional animation for each piece of clothing. Secondly, there is no texture.

[0004] There are two commonly used methods: one is to use somatosensory technology to achieve high-definition 2D clothing images naturally attached to the human body. At the Intel 2011 Digital Signage Annual Conference, Intel invited Kaioscar to demonstrate China's first officially mass-produced virtual fitting device K-MIRROR. The device provides high-definition clothing image preview and fitting, using Microsoft's KINECT somatosensory device, and a high-definition camera with a resolution of up to 1920X1080, 8-megapixel static photos, and Intel Core 2nd generation i5 CPU. The high-definition virtual fitting process is very smooth, and you can try on various costumes, opera costumes, and even anime COSPLAY costumes, which attracted many eyes. But the price is relatively expensive.

[0005] The other is a virtual fitting photo system. Users can choose clothing combinations through clothing materials prepared in advance by the APP. After selecting the clothing they are interested in, the user's head portrait is taken by the front camera in the designated shooting area displayed on the screen. The user completes the fitting experience by synthesizing the photos after shooting. Because the model and the clothes are real, the overall image can show the texture of the clothes to the maximum extent. The APP can also share the fitting photos to social networks through photo saving and one-click sharing functions. In addition to the basic functions of fitting, the APP has also added multiple functions such as changing hair and taking photos, which adds fun elements while realizing virtual fitting. However, since the body shapes of the model and the buyer are still different, the buyer will feel unreal after the synthesized photos, and the purchased clothes may not fit.

[0006] By using virtual reality technology to create a three-dimensional virtual model, users can try on clothes virtually, quickly and intuitively compare clothes of different sizes and design styles, and no longer need to spend a lot of time trying on clothes to effectively judge whether the fit and style of the clothes meet their needs. For some special types of clothing, such as ethnic costumes, which involve wearing scenes and complex decorations, a reasonably priced and practical device is needed. Summary of the invention

[0007] In view of the above-mentioned technical problems to be solved, the present invention provides a method for trying on 3D digital ethnic costumes, which gives ethnic costumes more presentation forms and brings users a more diverse trying on experience.

[0008] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0009] A method for trying on 3D digital ethnic costumes, comprising the following steps:

[0010] Step S1, generation of three-dimensional ethnic costume model

[0011] Based on 3D scanning and photography, the national costumes are scanned in all directions to obtain the body structure and data information of the costumes, and 3D modeling and three-dimensional models are used to generate three-dimensional national costume models;

[0012] Step S2: Post-processing of the 3D ethnic costume model

[0013] Based on the generated 3D ethnic costume model, dynamic simulation is performed, multi-level detail model is created, 3D ethnic costume model post-processing is completed, and it is integrated into the database;

[0014] Step S3, database construction

[0015] Based on deep learning and data collection, complete the collection and classification of ethnic costume data including three-dimensional ethnic costume models, introduction content, environmental background and rendering music, and generate a database;

[0016] Step S4: User identification

[0017] The depth camera identifies the user's position and posture to form a three-dimensional model of the user. The image processing software analyzes the depth camera data to capture the user's body contour and movements in real time.

[0018] Step S5: clothing selection

[0019] The user selects the ethnic costume to try on;

[0020] Step S6: Virtual try-on

[0021] The 3D ethnic costume model is rendered onto the user's image in real time, and the user's 3D model is fused with the 3D model of the selected costume to generate a 3D fitting model.

[0022] As a further improvement of the above technical solution:

[0023] Preferably, the step S1 specifically includes the following steps:

[0024] S1-1, conduct a full range of 3D scanning of ethnic costumes, use the scanning instrument to capture the point cloud data of the clothing surface, then integrate the point cloud data at different angles into a coordinate system through coordinate transformation and registration algorithm, and then remove redundant points by noise filtering and downsampling the point cloud data;

[0025] S1-2, by taking multiple photos at different angles, using image processing algorithms to detect and match feature points in the photos, using triangulation to calculate the position of the feature points in three-dimensional space, and then using a surface reconstruction algorithm to generate an initial model of a polygonal mesh from the point cloud;

[0026] S1-3, edit the initial model using 3D modeling tools to fix errors that occurred during scanning and reconstruction; then fix mesh cracks or polygon errors;

[0027] S1-4, formats the data from S1-1 to S1-3 into a compatible file format and stores it in a database.

[0028] Preferably, the step S2 specifically includes the following steps:

[0029] S2-1, establish the physical parameters of the cloth; divide the cloth into grids, where the grid nodes represent the mass points of the cloth, and the lines between the mass points represent the springs; calculate the force acting on each node at each simulation time step; detect the collision between the cloth and the human body model or other objects during the simulation process; the updated cloth shape and position at each time step are saved as the new model state;

[0030] S2-2, calculate the edge weights between each vertex of the initial model and its adjacent vertices; use the edge collapse algorithm to simplify the model mesh; generate models with different levels of detail based on the simplified mesh; dynamically select models for rendering based on viewing distance, viewing angle and performance requirements;

[0031] S2-3, integrate the generated models at different levels of detail into the database.

[0032] Preferably, the step S3 specifically includes the following steps:

[0033] S3-1, preprocessing the collected data, including data cleaning, standardization and formatting;

[0034] S3-2, based on the constructed data foundation, use image processing technology to extract features from the three-dimensional models or pictures of clothing, and classify all the three-dimensional ethnic clothing models to be tried on into the following three categories: ethnic classification, clothing type classification, and color classification;

[0035] S3-3, based on the classified three-dimensional ethnic costume model, complete the construction of the database model;

[0036] S3-4, front-end development and user interaction.

[0037] Preferably, the step S4 specifically includes the following steps:

[0038] S4-1, obtaining depth information through a depth camera, converting the depth information into a depth image, and obtaining three-dimensional point cloud data from the depth image;

[0039] S4-2, using the 3D point cloud data in the depth image to detect the key points of the human body and mark the key point positions, in a multi-camera system, reconstruct the 3D key points through images from multiple perspectives, and use the key point information to build a 3D skeleton model of the human body; use the state estimation method to track the posture and movement of the human body;

[0040] S4-3, generating a three-dimensional model of the user based on the depth data and the skeleton tracking data.

[0041] Preferably, the step S5 specifically includes the following steps:

[0042] S5-1, design a graphical user interface GUI, through which users can view and select different ethnic costumes;

[0043] S5-2, build background data management and clothing retrieval. The background server manages the database containing all ethnic clothing information. Clothing retrieval is to search and filter in the database to find clothing data that meets the conditions.

[0044] Preferably, the step S6 specifically includes the following steps:

[0045] S6-1, converting the vertices of the three-dimensional model from the object coordinate system to the screen coordinate system for rendering; in the rendering process, generating new geometry or adjusting existing geometry; converting the geometry in the three-dimensional space into pixels in the two-dimensional screen space, and applying a texture to each fragment (pixel);

[0046] S6-2, align the reference points of the clothing model and the user model through translation, rotation and scaling operations, and match the key points or reference points of the two; define the bone structure for the clothing model, so that each vertex of the clothing model is associated with one or more bones in the bone system; use the physics engine to simulate the cloth characteristics of the clothing, and the cloth simulation is based on the mass spring system; detect the collision between the clothing and the user model in real time;

[0047] S6-3, before rendering the clothing model onto the user's image, image processing and optimization are performed, and then the user's image and the clothing rendering are combined.

[0048] Compared with the prior art, the 3D digital ethnic costume try-on experience method provided by the present invention has the following advantages:

[0049] (1) The method for trying on 3D digital ethnic costumes of the present invention obtains the body shape information of users, and makes the virtual characters corresponding to the users try on virtual costumes and display them in a virtual scene. The virtual trying on experience is enriched with the environmental background and music background, and the user experience is improved.

[0050] (2) The 3D digital ethnic costume fitting experience method of the present invention breaks the original two-dimensional display mode, giving ethnic costumes more display forms while providing users with a more diverse fitting experience. DETAILED DESCRIPTION

[0051] The specific embodiments of the present invention are described in detail below. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0052] The method for trying on and experiencing 3D digital ethnic clothing of the present invention comprises the following steps:

[0053] Step S1, generation of three-dimensional ethnic costume model

[0054] Based on 3D scanning and photogrammetry technology, we conduct all-round scanning of ethnic costumes to obtain the structure and data information of the clothing body, and use 3D modeling and three-dimensional model optimization technology to generate a three-dimensional ethnic costume model. Specifically:

[0055] S1-1, full-scale 3D scanning of ethnic costumes

[0056] 3D scanning technology is the first step in generating a three-dimensional model, using a scanning device (such as a laser scanner, structured light scanner, or depth camera) to capture point cloud data of the object's surface.

[0057] There are three main scanning methods:

[0058] (1) Laser scanning: A laser scanner emits a laser beam onto the surface of an object, and the laser is reflected back to the sensor. The scanner measures the distance to the object based on the reflection time or phase change of the laser (time of flight method or phase method).

[0059] (2) Structured light scanning: A structured light scanner projects a series of known patterns (such as stripes of light or grids) onto the surface of an object, and the camera captures the deformation of these patterns. By comparing the projected pattern with the deformed pattern, the coordinates of each point in three-dimensional space are calculated.

[0060] (3) Depth camera (such as Kinect): Using infrared light or other light sources, depth cameras measure the light reflected from the surface of an object and combine it with computer vision algorithms to determine depth information.

[0061] The specific scanning method is:

[0062] S1-1-1, Data Capture

[0063] The scanner or camera moves on a predetermined path to capture point cloud data of the clothing surface. Point cloud data is a collection of three-dimensional coordinate points that represent the shape of the object surface.

[0064] The coordinates (X, Y, Z) of each point are calculated by distance measurement and scanning angle. For laser scanners, the calculation formula is:

[0065]

[0066] Where Z is the depth, c is the speed of light, and t is the time it takes for the laser to be emitted and returned.

[0067] S1-1-2, multi-view synthesis

[0068] In order to fully capture all the details of the garment, scans are usually taken from multiple angles.

[0069] The point cloud data at different angles are integrated into a unified coordinate system through coordinate transformation and registration algorithm. The least square method or iterative closest point (ICP) algorithm is used for registration, and the calculation formula is:

[0070]

[0071] Among them, E(R,T) is the error function, R is the rotation matrix, T is the translation vector, and p i and q i is the set of corresponding points.

[0072] S1-1-3, Data filtering and streamlining

[0073] The point cloud data was processed by noise filtering and downsampling to reduce the data volume and remove redundant points.

[0074] Use algorithms such as mean filtering or median filtering to remove noise points. Downsampling methods such as voxel grid filtering are used to reduce the density of points.

[0075] S1-2, constructing a three-dimensional model

[0076] Multiple high-resolution photos are used to reconstruct the 3D model through multi-view geometry and image registration.

[0077] By taking multiple photos at different angles, image processing algorithms are used to detect and match feature points in the photos. The positions of these feature points in three-dimensional space are calculated using triangulation. Specifically:

[0078] S1-2-1, feature point detection and matching

[0079] Feature points are extracted from each image using algorithms such as SIFT (Scale Invariant Feature Transform) or SURF (Speeded Up Robust Features).

[0080] Match the feature points in adjacent images to generate matching point pairs. The calculation formula for matching point pairs is:

[0081]

[0082] Among them, f i and f j are the feature point descriptors of the two images.

[0083] S1-2-2, Camera Pose Estimation and Point Cloud Reconstruction

[0084] Use camera calibration and relative pose estimation (such as the five-point method or the eight-point method) to estimate the camera position and orientation for each photo.

[0085] Based on the camera pose and feature point matching, the three-dimensional point cloud is calculated using the triangulation formula:

[0086]

[0087] Where P is the position of the 3D point, λ 1 and λ 2 is the scaling factor, K 1 and K 2 is the camera’s intrinsic parameter matrix, p 1 and p 2 are corresponding points on the image.

[0088] S1-2-3, Dense point cloud generation and surface reconstruction

[0089] Use a dense matching algorithm (such as PatchMatch) to generate a denser point cloud. Then use a surface reconstruction algorithm (such as Poisson reconstruction) to generate a polygonal mesh model from the point cloud.

[0090] S1-3, 3D modeling and 3D model optimization

[0091] After obtaining the point cloud or initial model, further processing and optimization are performed using 3D modeling software.

[0092] Use 3D modeling tools (such as Maya, Marvelous Designer) to edit the initial model to fix errors that may have occurred during the scanning and reconstruction process; then add details such as folds, textures and colors of clothing. Specifically:

[0093] S1-3-1, Mesh simplification and optimization

[0094] Use mesh simplification algorithms such as Quadric Error Metrics (QEM) to reduce the number of polygons while maintaining the shape and details of the model.

[0095] Calculate the error between each polygonal face, and then merge the faces with the smallest error. The formula is as follows:

[0096]

[0097] Where Q(v) is the error measure, a i ,b i ,c i ,d i are the plane parameters of each patch.

[0098] S1-3-2, texture mapping and material adjustment:

[0099] Texture mapping pastes a 2D image onto the surface of a 3D model using a UV coordinate system. UV coordinate mapping requires projecting the 3D points of the model onto a 2D plane.

[0100] Material adjustment includes the use of PBR (physically based rendering) technology to simulate real physical materials by adjusting parameters such as reflection, glossiness, and transparency.

[0101] S1-3-3, repair and detail enhancement:

[0102] Fix possible mesh cracks or polygon errors.

[0103] The model details are increased by subdivision surface. The calculation formula is as follows:

[0104]

[0105] Among them, V′ is the updated vertex position, α and β are weight coefficients, V i are the positions of adjacent vertices.

[0106] S1-4, Data integration and model generation

[0107] After completing 3D scanning, photogrammetry and model optimization, all data needs to be integrated into a unified 3D national costume model.

[0108] S1-4-1, point cloud to polygon mesh conversion

[0109] Convert point cloud data into polygonal meshes using Delaunay triangulation or Poisson reconstruction algorithms.

[0110] These algorithms generate triangles based on the proximity of the point cloud, ensuring smooth surfaces without holes.

[0111] S1-4-2, Data Formatting and Integration

[0112] All scanned and modeled data are formatted into compatible file formats (such as OBJ, FBX) for use in the virtual try-on system.

[0113] The generated models are stored in a database and associated with corresponding metadata (such as cultural background, historical significance, etc.).

[0114] In summary, in this step, the generation of the 3D ethnic costume model combines a variety of advanced technologies and computing logic, obtains the geometric data of the costume through 3D scanning and photogrammetry, uses 3D modeling software for editing and optimization, and finally generates a high-quality 3D model. The combination of these technologies ensures that the generated model is not only highly accurate, but also provides realistic visual effects and interactive experience.

[0115] Step S2: Post-processing of the 3D ethnic costume model

[0116] Based on the generated 3D ethnic costume model, Marvelous Designer software was used to perform dynamic simulation, create model versions with multiple levels of detail, complete 3D model post-processing, and integrate it into the database.

[0117] The post-processing of the 3D ethnic costume model mainly involves the use of Marvelous Designer software for dynamic simulation and the creation of multi-level of detail (LOD) models. This process aims to improve the realism of the model and optimize its performance in different application scenarios. The following is a detailed explanation of the principles used in this step and its calculation logic.

[0118] S2-1, dynamic simulation

[0119] Marvelous Designer is a 3D software for clothing design and simulation. Its power lies in its ability to simulate the physical properties of cloth, making clothing look and behave closer to reality in a virtual environment. Dynamic simulation is a major feature of the software, which allows clothing to move and flow naturally in a simulated physical environment.

[0120] Cloth physics simulation: By establishing the physical properties of cloth (such as density, elasticity, rigidity, friction coefficient, air resistance, etc.), simulation software can simulate the behavior of cloth under different conditions. These physical properties define how cloth moves under external forces such as gravity, wind, collision, etc.

[0121] Mass-Spring System: Dynamic simulation is usually based on a mass-spring system, which treats the cloth as a grid consisting of a large number of small mass points (nodes) and springs (edges) between these nodes. The mass points have mass and the springs have elastic coefficients.

[0122] Finite Element Analysis (FEA): In more complex simulations, finite element analysis is used to calculate the deformation of cloth under load. By dividing the cloth into many small finite elements (such as triangles or quadrilaterals), the FEA method can accurately simulate the stress and strain distribution of the cloth.

[0123] S2-1-1, Modeling of physical properties of cloth

[0124] Establish the physical parameters of the cloth, including density (ρ), elastic modulus (E), Poisson's ratio (ν), etc. These parameters determine the stiffness and flexibility of the cloth.

[0125] For example, the relationship between the elastic modulus (E) and Poisson's ratio (ν) describes the behavior of cloth when stretched and compressed through the following formula:

[0126] σ=E·∈

[0127] Here, σ is stress and ∈ is strain.

[0128] S2-1-2, Construction of Mass-Spring Model

[0129] The cloth is divided into grids, the grid nodes represent the mass points of the cloth, and the lines between the mass points represent the springs.

[0130] The restoring force of each spring is calculated according to Hooke's law:

[0131] F=-k·(LL 0 )

[0132] Where F is the spring force, k is the spring constant, L is the current spring length, and L 0 is the natural length of the spring.

[0133] S2-1-3, Force calculation and update

[0134] At each simulation time step, all forces acting on each node are calculated, including gravity (F g =m·g), air resistance, spring restoring force, and any other external forces (such as collision force).

[0135] Update the velocity and position of the node using Newton's second law:

[0136]

[0137] Where a is the acceleration, F total is the total force, m is the mass, and Δt is the time step.

[0138] S2-1-4, Collision Detection and Response

[0139] Detect collisions between cloth and mannequins or other objects during simulation.

[0140] Use a simple collision response model, such as adjusting node velocities after collision based on the reflected normal vector, to ensure that cloth does not penetrate other objects.

[0141] S2-1-5, simulation result output

[0142] The updated cloth shape and position at each time step are saved as the new model state.

[0143] The final output is the cloth shape after dynamic simulation, which can be used for rendering and display.

[0144] S2-2, Creating Multi-Level-of-Detail Models

[0145] Multiple level of detail (LOD) models are created to provide optimized performance and visual effects in different application scenarios (e.g., virtual try-on, virtual reality demonstration, etc.). LOD technology ensures a balance between visual effects and computational efficiency by creating different levels of detail for the model.

[0146] Level of Detail (LOD): The LOD of a model refers to multiple versions of the same model, each with different geometric complexity. Usually divided into three levels of detail: high, medium, and low.

[0147] Simplification algorithm: The creation of LOD models relies on mesh simplification algorithms, such as quadrilateral mesh simplification, vertex clustering, progressive meshes, and surface approximation. These algorithms automatically select appropriate LOD models for rendering based on viewing distance or computing resources.

[0148] S2-2-1, Initial model optimization

[0149] Start with a high-resolution model and optimize it according to the needs of the application scenario. This high-resolution model usually retains all geometric details and dynamic simulation results. Calculate the edge weights of each vertex of the model with its neighboring vertices. The edge weights can be based on the vertex normal, curvature, or color difference.

[0150] S2-2-2, mesh simplification

[0151] Use the edge collapse algorithm to simplify the model. The algorithm determines how to simplify the mesh by calculating the cost (i.e. error metric) of each edge collapse operation:

[0152]

[0153] Where E(v) is the vertex error metric, (a i ,b i ,c i ,d i ) are the coefficients of the plane equation.

[0154] The cheapest edges are collapsed first, and the mesh is adjusted to maintain a smooth surface.

[0155] S2-2-3, LOD model generation

[0156] Generate models with different levels of detail based on the simplified mesh. For example:

[0157] LOD 0: A complete, high-detail model, including all vertices and faces, typically used for close-up viewing.

[0158] LOD 1: Medium detail model with reduced vertex and face counts, for viewing at medium distances.

[0159] LOD 2: Low detail model, further simplifies the mesh, used for long-distance viewing or resource-constrained environments.

[0160] S2-2-4, viewing distance and performance optimization

[0161] Dynamically select the appropriate LOD model for rendering based on viewing distance, viewing angle, and performance requirements. Calculate the screen space error (Screen Space Error, SSE) for each model version to ensure that the visual error is not obvious when using low-detail models.

[0162] The SSE calculation formula is:

[0163]

[0164] Where E(v) is the vertex error metric, D is the view distance, and Z is the depth between the viewpoint and the object.

[0165] S2-2-5, Data integration into database

[0166] All generated LOD models and their associated metadata (such as materials, animation parameters, etc.) are integrated into the database for efficient use in subsequent virtual try-ons or displays.

[0167] S2-3, Database Integration

[0168] The generated 3D model (including LOD version) is integrated into the database to facilitate subsequent calling and display, and to ensure the efficiency and reliability of data management.

[0169] Relational database: used to store and manage 3D models and their related data (such as metadata, textures, animation sequences, etc.).

[0170] Index and query optimization: Improve data retrieval speed and system response efficiency by building indexes and optimizing queries.

[0171] S2-3-1, Data formatting and storage

[0172] Format all generated models and related data into standard file formats (such as OBJ, FBX, GLTF, etc.) and save them to the file system.

[0173] The database stores the model's metadata (such as model ID, LOD level, file path, texture mapping, physical properties, etc.).

[0174] S2-3-1, index creation

[0175] Create indexes for model data and metadata to speed up the retrieval process. For example, create indexes for fields such as model ID, LOD level, model category, etc.

[0176] S2-3-3, Data retrieval and loading

[0177] When a model needs to be loaded, the database is queried based on the requested LOD level and other parameters such as view distance, device performance, etc. Use query optimization techniques (such as hash indexes, B-trees, etc.) to quickly find the appropriate model and load it from the file system into memory for rendering.

[0178] S2-3-4, Data synchronization and update

[0179] Supports real-time update and version management of model data to ensure that the latest model data is used during virtual try-on and display.

[0180] In summary, in this step, the post-processing of the 3D ethnic costume model is done by using MarvelousDesigner software for dynamic simulation, creating a model version with multiple levels of detail, and integrating all data into the database. This process not only improves the realism and visual effects of the model, but also optimizes the performance of the model in different application scenarios. Advanced physical simulation, mesh simplification algorithms, and data management technologies are used to ensure that the generated model is of high quality and efficiency.

[0181] Step S3, database construction

[0182] Based on the needs of deep learning and data collection, data science and software engineering technologies are used to automatically collect and classify data on ethnic costumes, including three-dimensional models of ethnic costumes, introduction content, environmental background and rendering music, and generate a database.

[0183] The construction of the database involves multiple sub-steps (S3-1 to S3-4), which combine data science, software engineering, deep learning, machine learning, and database management techniques to build a comprehensive and rich ethnic costume database.

[0184] S3-1, Data acquisition and preprocessing

[0185] Based on the needs of deep learning and large-scale data collection, data science and software engineering technologies are used to complete the automated collection of data related to ethnic costumes, which includes comprehensive data on three-dimensional models, introduction content, environmental background, and rendered music.

[0186] Furthermore, the collected data are preprocessed, including data cleaning, standardization and formatting, to ensure the accuracy and consistency of the data and generate the basic construction of the database.

[0187] The data collection is to collect data related to ethnic costumes from various sources (such as literature databases, museum databases, online pictures and video resources, etc.) using automated tools (such as web crawlers, API calls), including three-dimensional models, introduction content, environmental background, and rendering music.

[0188] The data preprocessing is to clean, standardize and format the collected data to ensure the consistency and accuracy of the data. Data cleaning is to remove redundant, erroneous or incomplete data. Standardization is to unify the format and unit of data to ensure that all data are compatible and comparable. Formatting is to convert data into a structure suitable for database storage and query.

[0189] Furthermore, data collection is performed in the following ways:

[0190] (1) Web crawler: Use programming languages ​​such as Python to develop crawler scripts to automatically access and crawl the content of the target website. Use regular expressions and HTML parsing libraries (such as BeautifulSoup) to extract the required information from the web page.

[0191] (2) API call: Communicate with the data source (such as museum API, document database API) through the interface, send HTTP requests (GET, POST, etc.) to obtain data, and parse the data returned by the API using JSON or XML format.

[0192] (3) Automated data integration: Integrate data from different sources into an initial database or dataset (such as a Pandas DataFrame or SQL table) for subsequent processing.

[0193] Furthermore, data preprocessing includes the following steps:

[0194] S3-1-1, data cleaning

[0195] Check and handle missing data: Use interpolation, mean filling, deletion of missing values, etc. to fill or remove incomplete records.

[0196] De-duplicate data: Remove duplicate entries based on unique identifiers (such as model ID or clothing name).

[0197] S3-1-2, data standardization

[0198] Convert all data to a consistent format. For example, convert all lengths to meters, or convert all dates to the ISO standard (YYYY-MM-DD).

[0199] S3-1-3, Data Formatting

[0200] Convert data into a format suitable for storage and fast query. For example, convert 3D model data into standard file formats (such as OBJ, FBX), convert text data into UTF-8 encoded strings, and convert image and video data into standard formats such as JPEG and MP4.

[0201] S3-2, Data Classification

[0202] Based on the constructed data foundation, deep learning and machine learning techniques are used in combination with artificial intelligence methods to classify all three-dimensional ethnic costume models to be tried on.

[0203] It is divided into at least three categories. The first category is classified according to different ethnic groups, the second category is classified according to clothing types (such as double-breasted clothes, left-breasted clothes, right-breasted clothes, and cross-collared clothes), and the third category is classified according to color.

[0204] The classification method adopts deep learning and machine learning classification: deep learning (such as convolutional neural network, CNN) and machine learning algorithms (such as support vector machine SVM, K nearest neighbor algorithm KNN) are used to automatically classify the collected three-dimensional ethnic costume models.

[0205] Going a step further, artificial intelligence methods are used: combining image processing, feature extraction and pattern recognition techniques to analyze the visual characteristics of clothing (such as color, pattern, shape) to achieve automatic classification.

[0206] The specific steps include:

[0207] S3-2-1, training of data classification model

[0208] Feature extraction is performed first, and then the deep learning model is trained, using the back-propagation algorithm to update the network weights to minimize the loss function.

[0209] S3-2-1-1, Feature extraction: Use image processing technology to extract features from the three-dimensional model or picture of the clothing. For example, use edge detection algorithms (such as Sobel operator), color histograms and shape descriptors (such as Hu moments) to extract edge, color and shape features of clothing.

[0210] S3-2-1-2, Model training: Select a deep learning model (such as a convolutional neural network, CNN) for training. CNN extracts and classifies the features of the input image through multiple convolutional layers, pooling layers, and fully connected layers.

[0211] S3-2-1-3, input data (such as a 3D model rendering or a flat image of clothing) is converted into a feature vector that the model can understand. The training data set includes multiple categories of labels (such as ethnicity, clothing type, color), and the loss function (such as cross entropy loss) is used to calculate the error between the predicted and true labels.

[0212] Use the back-propagation algorithm to update the network weights to minimize the loss function:

[0213]

[0214] Among them, L is the loss function, y i is the true label, y i is the predicted probability.

[0215] S3-2-2, data classification

[0216] Data classification includes the following three categories: ethnic classification, clothing type classification, and color classification.

[0217] Ethnic classification: Based on the visual features of clothing (such as patterns and colors) and historical background information, a classification model is used to divide clothing into different ethnic categories (such as Han, Dong, Miao, etc.).

[0218] Clothing type classification: Based on the design and shape characteristics of clothing, a classification model is used to classify clothing into different types (such as double-breasted clothing, left-breasted clothing, right-breasted clothing, and cross-collared clothing).

[0219] Color classification: Use color recognition algorithms (such as color histogram matching or color clustering technology) to classify clothing by color.

[0220] S3-2-3, Model evaluation and optimization

[0221] Cross-validation: Divide the data into training set and validation set, and use the cross-validation method to evaluate the performance of the classification model to prevent overfitting.

[0222] S3-2-4, precision and recall calculation

[0223] Precision = TP / (TP+FP), Recall = TP / (TP+FN)

[0224] Among them, TP is true positive, FP is false positive, and FN is false negative.

[0225] S3-3, database model construction

[0226] Based on the classified three-dimensional ethnic costume model, the database model is constructed using a relational database management system;

[0227] The constructed database includes various types of data including three-dimensional ethnic costume models, introduction content, environmental background, and rendering music.

[0228] Relational Database Management System (RDBMS): Use a relational database (such as MySQL, PostgreSQL) to store and manage various types of data on ethnic costumes, including three-dimensional models, introduction content, environmental background, rendering music, etc.

[0229] Database modeling: Build a database model suitable for data storage and fast retrieval, including table structure design, index optimization, data constraints and relationship definition, etc.

[0230] The specific steps include:

[0231] S3-3-1, Database table design

[0232] Design multiple tables according to the data category (such as 3D model, clothing introduction, environment background, rendering music). Each table contains necessary fields (such as clothing ID, name, category, file path, description content, audio file path, etc.) and data types (such as string, integer, floating point number, BLOB, etc.).

[0233] Determine the relationship between tables (such as one-to-many, many-to-many) and connect them through foreign keys.

[0234] S3-3-2, Indexing and Optimization

[0235] Create indexes for frequently queried fields (such as clothing ID, category, ethnicity, color, etc.) to improve query speed.

[0236] Use normalization to reduce data redundancy and ensure database consistency and integrity. The third normal form (3NF) is often used to eliminate functional dependencies and ensure database structure optimization.

[0237] S3-3-3, Data Integrity and Constraints

[0238] Define primary key and foreign key constraints to ensure data uniqueness and referential integrity.

[0239] Set check constraints and default values ​​to maintain data validity and integrity.

[0240] S3-4, front-end development and user interaction

[0241] Based on Web technology and mobile application development, API development and front-end application creation strategies are adopted to complete the complete link from data collection to user interaction, ensuring that users can conveniently retrieve information in the database through an easy-to-use interface, and easily browse and try on different ethnic costumes based on the classification results.

[0242] The introduction to each clothing category includes historical background, production techniques, cultural significance, etc., and is displayed in various forms, such as text descriptions, pictures, videos, and 3D animations to enhance the user experience.

[0243] Web technology and mobile application development: Build user interaction interfaces based on web development technologies (such as HTML, CSS, JavaScript) and mobile application development frameworks (such as React Native and Flutter).

[0244] API development: Connect the front-end application with the database backend through RESTful API or GraphQL API to retrieve, update and interact with data.

[0245] The specific steps include:

[0246] S3-4-1, API development and integration

[0247] Design and implement RESTful API or GraphQL API for data communication between front-end and back-end. API supports CRUD operations (Create, Read, Update, Delete) to access and manage database data.

[0248] Use JSON or XML format to exchange data between client and server.

[0249] S3-4-2, front-end user interface design

[0250] Develop a responsive web interface or mobile app interface that allows users to easily browse and try on different ethnic costumes. The interface includes a categorized list of costumes, detail pages, search, and filtering capabilities.

[0251] Build dynamic and interactive user interfaces using JavaScript frameworks like React, Vue.js and UI libraries like Bootstrap, Material-UI.

[0252] S3-4-3, User Interaction Logic

[0253] Implement interaction logic based on user input (such as touch, click, slide, etc.), and manage application status and user data through event handlers and state management frameworks (such as Redux).

[0254] Provide rich multimedia content (such as text descriptions, images, videos, 3D animations) to enhance the user experience, and utilize the browser's multimedia API (such as HTML5 Video API) or the multimedia playback capabilities of mobile devices.

[0255] In summary, the database construction process in this step completes the automated collection, classification and database model construction of ethnic costume-related data through the combination of data science, software engineering, deep learning and machine learning technologies. The use of modern database technology and front-end development methods ensures that users can easily retrieve and browse information in the database, providing an easy-to-use virtual try-on platform that enhances user experience. Each sub-step of this process is based on specific principles and calculation logic to ensure the efficiency, accuracy and user-friendliness of the system.

[0256] Step S4: User identification

[0257] The user's position and posture are identified through a depth camera to form a three-dimensional model of the user. Using body tracking technology and image processing software to analyze camera data, the user's body contours and movements are captured in real time.

[0258] User identification involves using depth cameras and image processing technology to identify the user's position and posture, forming a 3D model of the user, and using body tracking technology to capture the user's body contours and movements in real time. This process combines multiple technologies such as computer vision, deep learning, human posture estimation, and 3D reconstruction. It includes the following steps:

[0259] S4-1, depth camera recognition

[0260] Depth camera: Depth cameras (such as Microsoft Kinect, Intel RealSense) measure the distance between an object and the camera by emitting infrared light or structured light and using a receiver to capture the reflected light. It can capture the depth information of an object in three-dimensional space and generate a depth image.

[0261] Depth image: A depth image is a two-dimensional image in which the value of each pixel represents the depth relative to the camera (usually in millimeters). The depth information is used to calculate the position of each pixel in three-dimensional space.

[0262] The specific steps include:

[0263] S4-1-1, deep data collection

[0264] Data acquisition: The depth camera emits infrared light or structured light patterns to the user, then receives the reflected light through the sensor, calculates the time of flight (ToF) of the light or the deformation of the structured light to obtain depth information.

[0265] Depth map generation: The calculated depth data is converted into a depth image (Depth Map), where the value of each pixel represents the distance (depth) from the point to the camera.

[0266] S4-1-2, point cloud data generation

[0267] Get 3D point cloud data from the depth image. A point cloud is a set of discrete points in a 3D coordinate system, where each point represents a part of the user's body surface. The 3D coordinates (X, Y, Z) of the point can be calculated from each pixel position (u, v) of the depth map and the corresponding depth value d:

[0268]

[0269] Among them, (c x ,c y ) is the principal point of the camera, (f x ,f y ) is the focal length of the camera.

[0270] S4-1-3, Deep data filtering and smoothing

[0271] Since depth data may contain noise and outliers, median filtering or mean filtering is used to smooth the depth image to remove noise and hole data and generate a more stable and accurate depth image.

[0272] S4-2, Human Body Tracking

[0273] Pose Estimation: By analyzing depth images or RGB images, the 3D coordinates of key points of the human body (such as head, shoulders, elbows, wrists, hips, knees, ankles, etc.) are estimated. These key points are used to build a human skeleton model.

[0274] Skeleton Tracking: Based on the human skeleton model, it tracks the user's posture and movements in real time. Skeleton tracking can not only identify the user's static posture (such as standing, sitting), but also track dynamic movements (such as walking, waving).

[0275] The specific steps include:

[0276] S4-2-1, key point detection

[0277] Depth image key point detection: Use the 3D point cloud data in the depth image to detect the key points of the human body. By identifying the characteristic parts of the human body (such as the head and the ends of the limbs) and using machine learning models (such as random forests or deep learning models) for classification, the key point locations are identified and labeled.

[0278] RGB image key point detection: In some application scenarios, RGB images are used in combination with depth images to detect key points on RGB images through convolutional neural networks (such as OpenPose, HRNet). These models are usually trained with large annotated datasets (such as COCO, MPII) and can accurately identify 2D key points of the human body.

[0279] S4-2-2, 3D key point inference

[0280] Monocular depth mapping: Use the depth image to directly obtain the depth information of each key point and convert the 2D image coordinates into 3D space coordinates.

[0281] Multi-view reconstruction: In a multi-camera system, 3D key points are reconstructed through images from multiple viewpoints to improve accuracy.

[0282] S4-2-3, skeleton model construction

[0283] The key point information is used to construct a 3D skeleton model of the human body. The skeleton model is composed of a series of joints (key points) and bones (line segments) connecting these joints.

[0284] The positions of joints are determined by the coordinates of 3D key points, and the skeleton is represented by line segments connecting adjacent joints. The skeleton model not only provides posture information of the human body, but can also be used for motion capture and posture analysis.

[0285] S4-2-4, posture and motion tracking

[0286] Kalman filter or particle filter: In order to smooth the position changes of key points and skeletons of the human body, state estimation methods (such as Kalman filter or particle filter) are used to track the posture and movement of the human body. These methods estimate the current state based on the state of the previous moment and the current observation data.

[0287] Motion prediction: Based on the historical position and speed of the human skeleton, the skeleton position of the next frame is predicted to improve the robustness and real-time performance of tracking.

[0288] S4-3, 3D model generation

[0289] 3D reconstruction: Generate a 3D model of the user based on depth data and skeleton tracking data. 3D reconstruction technology usually includes steps such as point cloud processing, surface reconstruction, and mesh generation.

[0290] Model fusion: Fusion of the user’s 3D model with the selected 3D clothing model to achieve a virtual try-on experience.

[0291] The specific steps include:

[0292] S4-3-1, Point cloud processing and surface reconstruction:

[0293] Point cloud filtering and downsampling: Use voxel grid filtering or statistical outlier removal to downsample and denoise the point cloud.

[0294] Surface Reconstruction: Generate a continuous 3D surface model from the processed point cloud using Poisson Surface Reconstruction or Convex Hull Algorithm.

[0295] Poisson surface reconstruction formula:

[0296]

[0297] Among them, φ is the implicit function and V is the gradient field of the point cloud.

[0298] S4-3-2, Grid Optimization:

[0299] Mesh Simplification: Use mesh simplification algorithms (such as Quadric Edge Collapse) to reduce the number of polygons to improve the rendering efficiency of the model.

[0300] Texture mapping: Generate texture coordinates for the 3D model so that the texture of the clothing blends seamlessly with the user's 3D model.

[0301] S4-3-3, Model Fusion

[0302] Geometric alignment: Geometrically align the user's 3D model with the clothing model so that the clothing can naturally adjust according to the user's posture and movements.

[0303] Physical simulation: Use a physical engine (such as NVIDIA PhysX) to simulate the natural flow and collision reaction of clothing on the user, enhancing the realism of trying on.

[0304] In summary, in this step, user identification uses a variety of advanced computer vision and deep learning technologies to obtain the user's three-dimensional information through a depth camera, estimate the user's posture and movement using a human body tracking algorithm, and generate a three-dimensional model of the user. The computational logic of this process involves steps such as deep image processing, key point detection and skeleton tracking, three-dimensional reconstruction, and model fusion. Through the comprehensive application of these technologies, the system can capture the user's movements in real time and accurately match virtual clothing to the user, achieving a realistic virtual try-on experience.

[0305] Step S5: clothing selection

[0306] The user selects the ethnic costumes to try on. The clothing selection mainly involves the user selecting the ethnic costumes they want to try on the system interface. This process combines user interface design, graphical user interface (GUI) technology, human-computer interaction technology, and background data management.

[0307] S5-1, Graphical User Interface (GUI)

[0308] Graphical User Interface (GUI): GUI is a visual interface for users to interact with the system. Through GUI, users can intuitively view and select different ethnic costumes. GUI usually consists of elements such as buttons, icons, menus, sliders, etc. Users can interact by touching, clicking, sliding, etc.

[0309] Responsive design: The GUI interface needs to respond in real time according to user operations, such as highlighting selected clothing, loading detailed clothing information, providing trial previews, etc.

[0310] User input processing: User input (such as touch, click, slide, voice command, etc.) is recognized and processed by the system, and then converted into specific operations (such as selecting clothing, viewing details, trying on, etc.).

[0311] The specific steps include:

[0312] S5-1-1, User interface design and implementation

[0313] Layout Design: Create a graphical user interface using HTML / CSS, front-end frameworks (such as React, Vue.js), or mobile development frameworks (such as Flutter, React Native) to display information such as thumbnails, names, and short descriptions of ethnic costumes.

[0314] Interactive element design: Provide an interactive element (such as a button or image area) for each piece of clothing on the interface, which users can click or touch to select clothing. Each element has a unique identifier (ID) corresponding to the clothing data.

[0315] S5-1-2, User input processing

[0316] Event listener: Set an event listener (such as onClick, onTouch events) for each interactive element to monitor the user's input actions. The event listener will be called when the user triggers an action (such as click, touch).

[0317] Event processing logic: When a user clicks or touches a certain clothing thumbnail, the event listener captures the event and calls the corresponding event processing function. The event processing function reads the ID of the clicked element and retrieves the corresponding clothing details from the database or memory based on the ID.

[0318] S5-1-3, Data loading and display

[0319] Asynchronous data request: If the clothing details and 3D model are not in the local cache, the event handler will load the details from the server through an asynchronous request (such as AJAX, Fetch API, or GraphQL request).

[0320] Data rendering: After loading is complete, the rendering mechanism of the front-end framework (such as React's setState or Vue's data binding) is used to display detailed information (such as the historical background, production process, cultural significance, 3D model, etc. of the clothing) on ​​the interface.

[0321] Interface updates: The interface dynamically updates to reflect the results of the user's selection, for example, highlighting the selected item, displaying item details, or loading a try-on preview model.

[0322] S5-1-4, User feedback mechanism

[0323] Visual feedback: Provide immediate feedback of user actions through highlighting, animation effects, etc. to confirm the user's selection. For example, a border or shadow effect may be added to the selected clothing thumbnail.

[0324] Audio feedback: Play clicks or sounds to confirm user actions, especially when tactile feedback is not obvious or visual feedback is insufficient.

[0325] Dynamic content loading: When a user selects a piece of clothing, the system can load the virtual try-on effect of the clothing in real time and display a preview of the effect after trying on on the screen.

[0326] S5-2, Backstage Data Management and Clothing Retrieval

[0327] Backend database management: The backend server of the system manages a database containing all ethnic costume information. The database stores the 3D model data, historical background, cultural significance, material information, color information, etc. of each costume.

[0328] Clothing search and filtering: Based on the clothing category selected by the user (such as ethnicity, clothing type, color, etc.), the system searches and filters in the database to quickly find clothing data that meets the conditions.

[0329] The specific steps include:

[0330] S5-2-1, Database Search

[0331] Query generation: When a user selects a piece of clothing, the system generates a database query statement (such as an SQL query) to retrieve data from the database based on the clothing's ID, category, ethnicity, color, and other conditions.

[0332] Query optimization: The database uses indexes and caches to optimize query speed. For example, indexes are created for frequently accessed fields (such as clothing ID, category, ethnicity, etc.) to improve data retrieval efficiency.

[0333] S5-2-2, Data Filtering

[0334] Conditional filtering: When performing a database query, you can set filtering conditions through the WHERE clause (such as SELECT * FROM clothing WHERE ethnicity = 'Dong' AND clothing type = 'double-breasted clothing') to return clothing data that meets the conditions.

[0335] Sorting and paging: Sort the query results according to the user's browsing needs (such as popularity, latest additions, user ratings, etc.) and display them in pages to improve the user experience.

[0336] S5-2-3, Data return and cache

[0337] Data caching: In order to reduce server load and response time, commonly used clothing data can be cached in the front end or middle layer (such as CDN, Redis cache). When the user selects, the data is loaded directly from the cache to avoid requesting the database every time.

[0338] Asynchronous processing: Use asynchronous data loading technology (such as AJAX, Fetch API) to ensure that the user interface will not be stuck due to data loading. Asynchronous processing technology can continue to respond to other user operations while loading data in the background.

[0339] S5-2-4, Data security and consistency

[0340] Data validation: When processing user requests, the system needs to verify the validity of user input and selection to prevent malicious operations or incorrect input. This can be achieved through front-end validation (such as JavaScript form validation) and back-end validation (such as regular expressions, database constraints, etc.).

[0341] Data consistency maintenance: Ensure that the clothing data selected by the user is consistent with the records in the database. If the clothing information in the database is updated, the system needs to be synchronized to the front-end interface in time to avoid inconsistencies.

[0342] S5-3, interactive feedback and optimization technology

[0343] User experience optimization: Improve the user experience of choosing clothing by providing an intuitive, easy-to-use interface and real-time feedback mechanism.

[0344] Optimize user operation paths: reduce the steps for users to search and select clothing, so that users can quickly and conveniently select the clothing they need.

[0345] The specific steps include:

[0346] S5-3-1, interface optimization

[0347] Dynamic filtering and search functions: Provide dynamic filtering and search boxes, so users can quickly find clothing of a specific ethnicity, color or clothing type. Use JavaScript or responsive programming of front-end frameworks to update the filtering results in real time.

[0348] Personalized recommendation system: Use machine learning algorithms (such as collaborative filtering and content recommendation) to analyze user choices and behaviors and recommend clothing that users may be interested in.

[0349] S5-3-2, Animation and Transition Effects

[0350] Transition animation: During the clothing selection process, use transition animation (such as fade in and fade out, sliding switching, etc.) to enhance the user experience and make the interface switching smoother and more natural.

[0351] Loading animation: During the data loading process, use loading animation (such as progress bar, spinning icon) to inform users that the background is processing the request, so as to prevent users from mistakenly thinking that the system is stuck or unresponsive.

[0352] S5-3-3, Equipment compatibility and adaptability

[0353] Responsive design: ensure that the interface is adaptively displayed on different devices (such as PC, tablet, mobile phone) to provide a consistent user experience.

[0354] Gesture recognition and voice control: Provide gesture recognition (such as sliding, pinching, and waving) and voice control functions on mobile devices or smart devices (such as AR / VR headsets) to enhance the diversity of user interaction.

[0355] In this step, clothing selection involves users interacting with the system through a graphical user interface to select the ethnic clothing they want to try on. This process combines modern front-end technology, back-end database management, data retrieval and filtering algorithms, human-computer interaction technology, etc., to achieve an intuitive and easy-to-use user selection interface and an efficient data retrieval mechanism. By optimizing the interface design and user experience, it ensures that users can quickly and conveniently complete clothing selection and try on.

[0356] Step S6: Virtual try-on

[0357] Through the virtual fitting engine, the 3D ethnic costume model is rendered onto the user's image in real time, and the user's 3D model is fused with the 3D model of the selected costume to generate a 3D fitting model.

[0358] In this step, the "virtual try-on" uses the virtual try-on engine to render the 3D ethnic costume model onto the user's image in real time, and fuses the user's 3D model with the 3D model of the selected costume to generate a realistic 3D try-on effect. This process involves a variety of computer graphics technologies and algorithms, including real-time rendering technology, model fusion technology, image processing and optimization technology, etc. It specifically includes the following steps:

[0359] S6-1, real-time rendering

[0360] Real-time rendering: Real-time rendering is a computing technology that generates images in real time during user interaction. It requires calculating and generating images of three-dimensional scenes in a short time (usually 60 frames per second or higher) to ensure that users can get a smooth and realistic visual experience when virtually trying on clothes.

[0361] Graphics Processing Unit (GPU) acceleration: Rendering is usually accelerated by a graphics processing unit (GPU), which has a large number of parallel processing units and is suitable for handling large amounts of graphics data and computing tasks.

[0362] Rendering pipeline: Real-time rendering is completed through the graphics rendering pipeline, which usually includes multiple stages such as vertex shading, geometry shading, fragment shading, rasterization, texture mapping, and lighting calculation.

[0363] The specific steps include:

[0364] S6-1-1, Vertex Shading

[0365] Vertex transformation: convert the vertices of the 3D model from the object coordinate system to the screen coordinate system. This process involves model transformation, view transformation, and projection transformation.

[0366] V=P·V·M·v

[0367] Among them, P is the vertex position, V is the model matrix, M is the view matrix, v is the projection matrix, and is the transformed vertex position.

[0368] S6-1-2, Geometry Shading

[0369] Geometry processing: During the rendering process, new geometry is generated or existing geometry is adjusted as needed (such as surface subdivision, hair rendering). This step is usually implemented by a geometry shader.

[0370] S6-1-3, Rasterization

[0371] Rasterization: Converts geometry in three-dimensional space into pixels in two-dimensional screen space. The rasterization stage interpolates the triangle vertices to calculate the position and color of each pixel.

[0372] Depth testing and blending: Depth testing is performed using the depth buffer (Z-buffer) to determine whether a pixel is in front of the line of sight and therefore whether it should be rendered.

[0373] S6-1-4, Fragment Shading

[0374] Texture mapping: Apply a texture to each fragment (pixel). Texture mapping is to map a texture image onto the surface of a 3D model, making the model look more realistic.

[0375] Color = Texture(u,v)

[0376] Among them, (u,v) is the texture coordinate, and Texture(u,v) is the color value sampled by the texture.

[0377] Lighting calculation: Use lighting models (such as Phong lighting model, Blinn-Phong lighting model, PBR lighting model) to calculate the lighting effects of the fragment, including diffuse reflection, specular reflection, and ambient light.

[0378] I=k a I a +k d (L·N)I d +k s (R.V) n I s

[0379] Where I is the final color value of the pixel, k a , k d ,k s are the ambient, diffuse and specular reflection coefficients, I a ,I d ,I s are the ambient light, diffuse light and specular light intensities, L is the light source direction, N is the normal direction, R is the reflection direction, V is the sight direction, and n is the highlight index.

[0380] S6-1-5, Anti-aliasing and post-processing

[0381] Anti-Aliasing: Use Multi-Sampling Anti-Aliasing (MSAA) or Fast Approximate Anti-Aliasing (FXAA) algorithms to smooth edges and reduce jagged effects.

[0382] Post-processing effects: Apply post-processing effects (such as HDR, color correction, depth of field, motion blur, etc.) to enhance visual effects.

[0383] S6-2, Model Fusion

[0384] Model fusion: Model fusion is to merge the user's 3D human body model with the selected 3D clothing model so that the clothing can adapt to the user's body shape and posture. The model fusion process includes technologies such as geometric alignment, bone binding, and cloth simulation.

[0385] Geometric alignment: Geometric alignment refers to aligning the user's 3D model with the clothing model in space so that the clothing fits closely to the user's body.

[0386] Bone binding and skinning: In order to achieve natural deformation of clothing when the user's body moves, the clothing model is usually bound to the user's bones, and the deformation of the clothing is controlled by skin weights.

[0387] The specific steps include:

[0388] S6-2-1, Geometric Alignment

[0389] Coordinate alignment: Align the clothing model with the reference points of the user model (such as waist, shoulder, neck) through translation, rotation and scaling operations. Geometric alignment can be achieved by matching key points or reference points of the two.

[0390] ICP algorithm (Iterative Closest Point): When more precise alignment is required, use the iterative closest point algorithm (ICP) to optimize alignment accuracy: The ICP algorithm calculates the optimal rigid transformation (rotation and translation) by minimizing the Euclidean distance between the source point cloud and the target point cloud.

[0391]

[0392] Among them, E(R, T) is the error function, R is the rotation matrix, T is the translation vector, and p i and q i are matching pairs of points.

[0393] S6-2-2, Skeleton Binding and Skinning Calculation

[0394] Bone binding: Define the bone structure for the clothing model so that each vertex of the clothing model is associated with one or more bones in the bone system. Usually a skeletal animation system (such as the dual quaternion skinning algorithm, Dual QuaternionSkinning) is used to handle skin deformation.

[0395] Skin weight calculation: Skin weights determine how each vertex is affected by different bones. The final position of each vertex is determined by the positions and weights of all influencing bones:

[0396]

[0397] Among them, v' is the position of the vertex after deformation, v is the original vertex position, and w i is the bone weight, B i It is the bone transformation matrix S6-2-3, cloth simulation

[0398] Physics simulation: Use physics engines (such as NVIDIA PhysX, Havok Cloth) to simulate the cloth properties of clothing, ensuring that clothing can flow and deform naturally based on the user's movements.

[0399] Mass-Spring System: Cloth simulation is usually based on a mass-spring system, which treats cloth as a grid consisting of a large number of small mass points and springs between these mass points. The physical properties of each spring (such as stiffness, damping) control the dynamic behavior of the cloth:

[0400] F=-k·(LL 0 )-c·v

[0401] Where F is the spring force, k is the spring constant, L is the current spring length, and L 0 is the natural length, c is the damping coefficient, and v is the velocity.

[0402] S6-2-4, Collision Detection and Response

[0403] Collision detection: Real-time detection of collisions between clothing and user models to ensure that clothing does not penetrate the user model. Commonly used collision detection algorithms include AABB (axis-aligned bounding box), OBB (orientation bounding box), Sphere Bounding, etc.

[0404] Collision response: When a collision is detected, the system needs to calculate the collision response and adjust the vertex position of the clothing to avoid penetration. Commonly used collision response technologies include normal vector-based rebound, depth correction, etc.

[0405] S6-3, Image Processing and Optimization

[0406] Image processing and optimization: Before rendering the clothing model onto the user's image, a series of image processing and optimization operations are required to ensure the realism and rendering efficiency of the final effect.

[0407] Deep blending and synthesis: Deep blending technology is used to seamlessly combine the user's image and the rendering results of the clothing to avoid visual conflicts.

[0408] The specific steps include:

[0409] S6-3-1, Deep Mixing

[0410] Depth buffer synthesis: Use the depth buffer (Z-buffer) to determine the rendering order of each pixel. The rendering results of the user's 3D model and clothing model are stored in different depth buffers, and the final display content of each pixel is determined by comparing the depth values.

[0411]

[0412] Among them, Z 服饰 and Z 用户 is the depth value of clothing and user model, Color final is the final displayed color value.

[0413] S6-3-2, Image Optimization

[0414] Color Correction: Adjust image brightness, contrast, and saturation to match real-world lighting conditions and the user’s imaging style.

[0415] Image filtering: Apply image filtering (such as Gaussian blur, sharpening filter) to enhance image details or soften overly sharp edges.

[0416] S6-3-3, performance optimization

[0417] Graphics hardware acceleration: Use the parallel computing power of the GPU to accelerate rendering and physical calculations to ensure real-time performance. Reduce GPU load by optimizing shader code and using low-polygon models.

[0418] Dynamic detail adjustment: Dynamically adjust rendering details and resolution (Level of Detail, LOD) according to the performance of the user's device and the complexity of the current scene to ensure a smooth user experience.

[0419] In summary, in this step, virtual try-on realizes the real-time rendering of the 3D ethnic costume model and the fusion of the user's 3D model through the virtual try-on engine. This process uses a variety of advanced computer graphics and computer vision technologies, including real-time rendering, model fusion, cloth simulation, image processing and optimization. Through the comprehensive application of these technologies, the system can generate a highly realistic virtual try-on effect and provide users with an immersive try-on experience.

[0420] The above implementation cases are only preferred embodiments of the present invention and are not intended to limit the present invention in any form. Although the present invention has been disclosed as above with preferred embodiments, they are not intended to limit the present invention. Therefore, any simple modification, equivalent changes and modifications made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for trying on 3D digital ethnic costumes, characterized in that: The following steps are involved: Step S1, generation of three-dimensional ethnic costume model Based on 3D scanning and photography, the national costumes are scanned in all directions to obtain the body structure and data information of the costumes, and 3D modeling and three-dimensional models are used to generate three-dimensional national costume models; Step S2: Post-processing of the 3D ethnic costume model Based on the generated 3D ethnic costume model, dynamic simulation is performed, multi-level detail model is created, 3D ethnic costume model post-processing is completed, and it is integrated into the database; Step S3, database construction Based on deep learning and data collection, complete the collection and classification of ethnic costume data including three-dimensional ethnic costume models, introduction content, environmental background and rendering music, and generate a database; Step S4: User identification The depth camera identifies the user's position and posture to form a three-dimensional model of the user. The image processing software analyzes the depth camera data to capture the user's body contour and movements in real time. Step S5: clothing selection The user selects the ethnic costume to try on; Step S6: Virtual try-on The 3D ethnic costume model is rendered onto the user's image in real time, and the user's 3D model is fused with the 3D model of the selected costume to generate a 3D fitting model.

2. The method for trying on 3D digital ethnic costumes according to claim 1, characterized in that: The step S1 specifically includes the following steps: S1-1, conduct a full range of 3D scanning of ethnic costumes, use the scanning instrument to capture the point cloud data of the clothing surface, then integrate the point cloud data at different angles into a coordinate system through coordinate transformation and registration algorithm, and then remove redundant points by noise filtering and downsampling the point cloud data; S1-2, by taking multiple photos at different angles, using image processing algorithms to detect and match feature points in the photos, using triangulation to calculate the position of the feature points in three-dimensional space, and then using a surface reconstruction algorithm to generate an initial model of a polygonal mesh from the point cloud; S1-3, edit the initial model using 3D modeling tools to fix errors that occurred during scanning and reconstruction; then fix mesh cracks or polygon errors; S1-4, formats the data from S1-1 to S1-3 into a compatible file format and stores it in a database.

3. The method for trying on 3D digital ethnic costumes according to claim 2, characterized in that: The step S2 specifically includes the following steps: S2-1, establish the physical parameters of the cloth; divide the cloth into grids, where the grid nodes represent the mass points of the cloth, and the lines between the mass points represent the springs; calculate the force acting on each node at each simulation time step; detect the collision between the cloth and the human body model or other objects during the simulation process; the updated cloth shape and position at each time step are saved as the new model state; S2-2, calculate the edge weights between each vertex of the initial model and its adjacent vertices; use the edge collapse algorithm to simplify the model mesh; generate models with different levels of detail based on the simplified mesh; dynamically select models for rendering based on viewing distance, viewing angle and performance requirements; S2-3, integrate the generated models at different levels of detail into the database.

4. The method for trying on and experiencing 3D digital ethnic costumes according to claim 3 is characterized in that: The step S3 specifically includes the following steps: S3-1, preprocessing the collected data, including data cleaning, standardization and formatting; S3-2, based on the constructed data foundation, use image processing technology to extract features from the three-dimensional models or pictures of clothing, and classify all the three-dimensional ethnic clothing models to be tried on into the following three categories: ethnic classification, clothing type classification, and color classification; S3-3, based on the classified three-dimensional ethnic costume model, complete the construction of the database model; S3-4, front-end development and user interaction.

5. The method for trying on and experiencing 3D digital ethnic costumes according to claim 4, characterized in that: The step S4 specifically includes the following steps: S4-1, obtaining depth information through a depth camera, converting the depth information into a depth image, and obtaining three-dimensional point cloud data from the depth image; S4-2, using the 3D point cloud data in the depth image to detect the key points of the human body and mark the key point positions, in a multi-camera system, reconstruct the 3D key points through images from multiple perspectives, and use the key point information to build a 3D skeleton model of the human body; use the state estimation method to track the posture and movement of the human body; S4-3, generating a three-dimensional model of the user based on the depth data and the skeleton tracking data.

6. The method for trying on and experiencing 3D digital ethnic costumes according to claim 5, characterized in that: The step S5 specifically includes the following steps: S5-1, design a graphical user interface GUI, through which users can view and select different ethnic costumes; S5-2, build background data management and clothing retrieval. The background server manages the database containing all ethnic clothing information. Clothing retrieval is to search and filter in the database to find clothing data that meets the conditions.

7. The method for trying on and experiencing 3D digital ethnic costumes according to claim 6, characterized in that: The step S6 specifically includes the following steps: S6-1, converting the vertices of the three-dimensional model from the object coordinate system to the screen coordinate system for rendering; in the rendering process, generating new geometry or adjusting existing geometry; converting the geometry in the three-dimensional space into pixels in the two-dimensional screen space, and applying a texture to each fragment (pixel); S6-2, align the reference points of the clothing model and the user model through translation, rotation and scaling operations, and match the key points or reference points of the two; define the bone structure for the clothing model, so that each vertex of the clothing model is associated with one or more bones in the bone system; use the physics engine to simulate the cloth characteristics of the clothing, and the cloth simulation is based on the mass spring system; detect the collision between the clothing and the user model in real time; S6-3, before rendering the clothing model onto the user's image, image processing and optimization are performed, and then the user's image and the clothing rendering are combined.

Images