Real-time commodity try-on video generation system and method based on virtual modeling

Abstract

The invention discloses a real-time commodity try-on video generation system and method based on virtual modeling, and relates to the technical field of computer graphics, and the method comprises the steps: fusing a spatial illumination balance degree NG, a reflection response matching factor AM, and a color difference Dc and an edge blurring rate Nm of a rendering result and a real image through a visual reality index EV; and calculating and outputting a visual truth index EV, setting a visual quality threshold Eth according to a visual psychology JND model, carrying out secondary comparison evaluation on the visual quality threshold Eth and the visual truth index EV, and constructing a complete visual fitting evaluation system. When a rendering result is lower than a threshold value, the system automatically starts an intelligent intervention mechanism which comprises sampling frame rate improvement, local shadow enhancement and material parameter dynamic adjustment, and a set of closed-loop feedback path from acquisition to evaluation to reconstruction is constructed. According to the mechanism, rendering adaptive correction can be realized on the premise that hardware does not need to be replaced, and the robustness, universality and user experience satisfaction of the system are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics technology, and in particular to a system and method for generating real-time product try-on videos based on virtual modeling. Background Art

[0002] The present invention relates to the field of virtual reality and human-computer interaction technology, and more specifically to computer vision-driven visualization technology for virtual goods. In particular, it relates to a method for generating real-time product try-on videos based on virtual modeling. This method dynamically fuses a user's real-time video image with a virtual three-dimensional model of the product to achieve virtual presentation and visual mapping of clothing, glasses, accessories, and other products on different users. This invention is widely used in new retail models such as online shopping scenarios, live virtual try-ons, and 3D product displays, and has the technical advantages of strong real-time performance, high immersion, and good personalization.

[0003] At present, existing virtual try-on technologies usually use a three-dimensional rendering mechanism under fixed light source conditions, which fails to dynamically match the actual lighting environment of the user. As a result, the generated try-on images have problems such as inconsistent lighting and reflections, serious deviations in material performance, and missing occlusions and shadows. Especially in multi-user, multi-environment, and multi-device scenarios, the current try-on system lacks flexibility and cannot adjust material performance in a timely manner according to the different ambient light conditions and angles of incidence of users. As a result, the generated videos show unnatural reflections on clothing, poor fit, and visual dislocation. In addition, the system lacks an objective evaluation and feedback mechanism for the rendering effect and cannot effectively identify "rendering failure" scenes, which seriously affects the user's try-on experience and the visual restoration of the product.

[0004] These issues primarily stem from the existing system's lack of perception and control over differences in ambient lighting and material reaction mechanisms. Because the user's location may present uneven lighting, extreme angles of incidence, and significant differences in brightness between the face and body, virtual clothing materials struggle to maintain correct reflective logic and shadow projection, resulting in unusual visual effects such as floating clothing, inconsistent texture mapping, and deviating colors from reality. This deviation not only affects users' perception of the product's true texture but also undermines product trust and purchase conversion rates on e-commerce platforms. It can even lead to widespread misinterpretations of false displays in social media sharing scenarios, hindering the commercialization of virtual try-on technology and the building of user trust. Summary of the Invention

[0005] In response to the deficiencies of the prior art, the present invention provides a system and method for generating real-time product try-on videos based on virtual modeling, which solves the problems mentioned in the background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: comprising the following steps:

[0007] S1. Collect video data of users trying on clothes through visual sensors, and transmit the video data to the virtual model for computer vision and image processing to obtain an ambient light vector set. Then, pre-process the ambient light vector set to obtain a standardized data set.

[0008] S2. Based on the standardized data set, calculate and output the spatial illumination balance NG, set the illumination uniformity interval threshold, perform preliminary comparative evaluation with the illumination balance NG, divide the illumination levels, and output the level label LG;

[0009] S3. Calculate and output the reflection response matching factor AM of the virtual material based on the grade label LG, and redefine the reflection condition of the product material;

[0010] S4. Calculate and output the visual realism index EV through the reflection response matching factor AM and the spatial illumination balance NG, set the visual quality threshold Eth and the visual realism index EV for secondary comparative evaluation, and trigger the reconstruction rendering strategy based on the evaluation results.

[0011] Preferably, said S1 includes S11, S12 and S13;

[0012] S11, setting the forehead, chest, and knees of the user trying on clothes as virtual collection points, and using a visual sensor to collect color video frames of the collection points to obtain video data of the user;

[0013] S12. directly connecting the visual sensor to the virtual model using a direct line connection, transmitting the acquired video data to the virtual model, and performing computer vision and image processing on the video data in the virtual model;

[0014] The computer vision and image processing method uses the OpenCV library to load video data, extracts each frame in the video data, obtains a frame image, converts the frame image into a grayscale image, and uses MediaPipe and OpenPose to detect human key points on the grayscale image, identifies the three human key points of the forehead, chest and knees in the frame image, and sets them as the corresponding ROI area of ​​interest;

[0015] S13. Based on the ROI point of interest region in the grayscale image, feature extraction is performed to obtain an ambient lighting vector set, and the ambient lighting vector set is subjected to Z-Score normalization processing to eliminate the dimensionality influence of all parameters in the ambient lighting vector set, thereby obtaining a standardized data set.

[0016] The standardized data set includes the light intensity I of the i-th collection point i , the angle J between the incident light at the i-th collection point and the surface normal iand the illumination gradient difference Gz between the i-th collection point and the j-th collection point ij ;

[0017] The light intensity I of the i-th collection point i Extract the mean value of the pixel values ​​within the ROI interest area in the grayscale image;

[0018] The angle J between the incident light at the i-th collection point and the surface normal i By using the DeepLearning-based method to generate a 3D human body model, the normal vectors of all acquisition points are obtained. The light source direction is inferred from the highlight points in the frame image through the Phong reflection model to obtain the light source direction vector. The angle between the light source direction vector and the normal vectors of all acquisition points is calculated using the dot product formula.

[0019] The illumination gradient difference Gz between the i-th collection point and the j-th collection point ij Based on the light intensity I of the collection point i i and the angle J between the incident light and the surface normal at the i-th collection point i Perform gradient difference calculation to obtain the illumination gradient difference between the i-th collection point and the j-th collection point, and perform average extraction on each collection point i and all collection points.

[0020] Preferably, said S2 includes S21 and S22;

[0021] S21. Based on the standardized data set, calculate and output the spatial illumination balance NG, which measures the uniformity and consistency of illumination received by the entire user's ROI area of ​​interest;

[0022] The physical meaning of the formula lies in that during virtual try-on, different parts of the user's body, such as the forehead, chest, and knees, may experience obvious uneven lighting due to issues such as the direction of the light source and environmental occlusion. We must not only consider brightness, but also the incident direction of the light source, the orientation of the material surface, that is, the normal vector, and the degree of light and dark changes in the spatial distribution. The product of the effective illumination difference and the spatial unevenness factor is used to express the consistency of the whole-body lighting, and then perform mean calculations to analyze uniformity.

[0023] Preferably, in S22, based on the tolerance range of perceptible differences of the human eye, a lighting uniformity interval threshold is set, wherein the lighting uniformity interval threshold includes a first lighting threshold F1 and a second lighting threshold F2. A preliminary comparative evaluation is performed on the spatial lighting balance NG obtained in real time and the lighting uniformity interval threshold to determine the uniformity of the ambient light distribution of the user trying on the wearer, and the current user trying on the wearer is divided into different levels based on the preliminary comparative evaluation results. The specific evaluation content is as follows;

[0024] When the spatial illumination balance NG ≥ the first illumination threshold F1, it indicates that the ambient illumination distribution is uniform and consistent. In this case, it is classified as G1 level. The original material parameters are used without disturbance and standard rendering is performed directly.

[0025] When the second illumination threshold F2 ≤ spatial illumination balance NG < the first illumination threshold F1, it indicates that there is partial illumination deviation and local vignetting, and the lighting is classified as level G2.

[0026] When the spatial illumination balance NG is less than the second illumination threshold F2, it indicates that the ambient light has abnormal contrast, the environment is asymmetric, the incident angle is chaotic, and the ambient light is bad. In this case, it is classified as G3 level.

[0027] The G1 level, the G2 level, and the G3 level are aggregated to output a level label LG, wherein the specific form of the level label LG is LG∈{G1,G2,G3}.

[0028] Preferably, said S3 includes S31 and S32;

[0029] S31. Based on the level label LG, it is converted into a dynamic disturbance control factor RD (LG), and its specific form is set as:

[0030] .

[0031] Preferably, S32, based on the dynamic disturbance control factor RD (LG) combined with the basic reflectivity R0 of the clothing material and the angle &J between the incident light and the surface normal at the main collection point, calculate the output reflection response matching factor AM, and dynamically disturb the material reflection according to the environment level.

[0032] Preferably, said S4 includes S41, S42 and S43;

[0033] S41. After dynamically disturbing the material reflection, render the ROI area of ​​interest of the user trying on the clothes to obtain a video rendering frame image. Then, perform a difference calculation on the average pixel value of the ROI area of ​​interest of the video rendering frame image and the real shot frame image, and output the color difference Dc between the rendering result and the real image;

[0034] Use the Mask semantic segmentation output mask to extract the edge area of ​​the material of the clothing being tried on in the rendered frame image. Apply the Sobel gradient operator to the grayscale image of the edge area to extract the gradient amplitude. Calculate the ratio of the standard deviation of the gradient amplitude to the upper limit of the gradient amplitude to obtain the acquisition edge blur rate Nm.

[0035] The color difference Dc and edge blur rate Nm between the rendering result and the real image are normalized using Z-Score to eliminate the dimension effect;

[0036] The reflection response matching factor AM and the spatial illumination balance NG are combined with the color difference Dc and edge blur rate Nm between the rendering result and the real image to calculate and output the visual realism index EV, which measures the consistency between the rendering result and the real picture.

[0037] Preferably, in S42, a visual quality threshold Eth is set based on a visual psychology JND model, and a secondary comparative evaluation is performed on the visual reality index EV obtained in real time to determine the eligibility of the current rendering of the user trying on clothes. The specific evaluation content is as follows;

[0038] When the visual realism index EV ≥ the visual quality threshold Eth, it means rendering and,maintaining the current strategy;

[0039] When the visual realism index EV is less than the visual quality threshold Eth, it means that the rendering is unqualified, and the intelligent intervention strategy is activated.

[0040] Preferably, S43, after the rendering fails, starting an intelligent intervention strategy, wherein the intelligent intervention strategy includes visual sensor intelligent intervention, shadow intelligent intervention and material disturbance intelligent intervention;

[0041] The sensor intelligent intervention increases the current sampling frame rate by 50%;

[0042] The shadow intelligent intervention renders virtual shadows by highlighting points in the ROI area of ​​interest;

[0043] The material disturbance intelligent intervention automatically adjusts the material disturbance sensitivity coefficient in the visual realism index EV to Increase by 50%, while affecting the attenuation factor of the light Reduced by 50%.

[0044] A real-time product try-on video generation system based on virtual modeling, including an environmental parameter acquisition and processing module, a spatial lighting balance analysis module, a grade label output module, and a rendering result analysis module;

[0045] The environmental parameter acquisition and processing module collects video data of users trying on clothes through visual sensors, and transmits the video data to the virtual model for computer vision and image processing to obtain an environmental lighting vector set, and then pre-processes the environmental lighting vector set to obtain a standardized data set;

[0046] The spatial illumination balance analysis module calculates and outputs the spatial illumination balance NG based on the standardized data set, sets the illumination uniformity interval threshold, performs preliminary comparative evaluation with the illumination balance NG, divides the illumination level, and outputs the level label LG;

[0047] The grade label output module calculates and outputs the reflection response matching factor AM of the virtual material based on the grade label LG, thereby redefining the reflectivity of the commodity material.

[0048] The rendering result analysis module calculates and outputs the visual realism index EV through the reflection response matching factor AM and the spatial illumination balance NG, sets the visual quality threshold Eth and the visual realism index EV for secondary comparative evaluation, and triggers the reconstruction rendering strategy based on the evaluation results.

[0049] The present invention provides a system and method for generating real-time product try-on videos based on virtual modeling. It has the following beneficial effects:

[0050] (1) This method sets the user's forehead, chest, and knees as collection points, integrates grayscale pixel brightness, normal incidence angle, and illumination gradient difference, constructs spatial illumination balance NG, and automatically divides the environment into different levels based on illumination consistency. Compared with existing virtual try-on systems that rely solely on brightness or single-point light source estimation, this method can dynamically quantify spatial illumination differences, effectively improving the ability to adapt to complex asymmetric lighting environments, so that clothing presents a natural and consistent lighting state on all parts of the user's body, thereby significantly improving visual fit and realism.

[0051] (2) This method sets the dynamic perturbation control factor RD (LG) based on the divided level label LG, and outputs the reflection response matching factor AM by combining the basic reflectivity R0 of the clothing material and the angle &J between the incident light and the surface normal at the main acquisition point. This reflection response matching factor AM can actively enhance the change of the reflection curve when the lighting is uneven, thereby enhancing the material's sensitivity to the real lighting environment. Compared with the traditional method of using static mapping, this method can dynamically adjust the reflection performance, especially under strong light, backlight or local dark corner conditions, effectively avoiding material mapping and fake face performance, and greatly improving the naturalness and realism of virtual clothing in different environments.

[0052] (3) This method uses the visual realism index EV to integrate the spatial illumination balance NG, the reflection response matching factor AM, the color difference Dc between the rendering result and the real image, and the edge blur rate Nm to calculate and output the visual realism index EV. Based on the visual psychology JND model, the visual quality threshold Eth is set and the visual realism index EV is used for secondary comparative evaluation to build a complete visual fitting evaluation system. When the rendering result is lower than the threshold, the system automatically activates the intelligent intervention mechanism, including increasing the sampling frame rate, enhancing local shadows, and dynamically adjusting material parameters, to build a closed-loop feedback path from acquisition to evaluation to reconstruction. This mechanism can achieve adaptive rendering correction without replacing hardware, greatly improving the robustness, versatility and user experience satisfaction of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of the steps of the method for generating real-time product try-on videos based on virtual modeling according to the present invention;

[0054] Figure 2 This is a flow chart of a real-time product try-on and try-on video generation system based on virtual modeling according to the present invention;

[0055] Figure 3 Schematic diagram of ROI area of ​​interest. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] Example 1

[0058] See also Figure 1 and Figure 3 The present invention provides a method for generating a real-time product try-on video based on virtual modeling. To achieve the above purpose, the present invention is implemented through the following technical solutions: comprising the following steps:

[0059] S1. Collect video data of users trying on clothes through visual sensors, and transmit the video data to the virtual model for computer vision and image processing to obtain an ambient light vector set. Then, pre-process the ambient light vector set to obtain a standardized data set.

[0060] S2. Based on the standardized data set, calculate and output the spatial illumination balance NG, set the illumination uniformity interval threshold, perform preliminary comparative evaluation with the illumination balance NG, divide the illumination levels, and output the level label LG;

[0061] S3. Calculate and output the reflection response matching factor AM of the virtual material based on the grade label LG, and redefine the reflection condition of the product material;

[0062] S4. Calculate and output the visual realism index EV through the reflection response matching factor AM and the spatial illumination balance NG, set the visual quality threshold Eth and the visual realism index EV for secondary comparative evaluation, and trigger the reconstruction rendering strategy based on the evaluation results.

[0063] In this embodiment, in S1, the method captures video frame data by setting virtual collection points at key areas of the user's body, such as the forehead, chest, and knees. The data is then processed for grayscale, key point detection, and feature extraction to construct an ambient lighting vector set. This lighting data is then Z-score normalized to obtain uniform dimensional illumination intensity, incident angle, and spatial gradient differences, establishing a standardized dataset for subsequent calculations. In S2, based on this dataset, a spatial lighting balance index NG for the user's current environment is calculated and output. An evaluation threshold is set based on the human eye's perceptible tolerance range, and the user's current environment is graded to generate a grade label LG. In S3, the dynamic perturbation control factor RD(LG) is dynamically adjusted based on the grade label LG. The base reflectance R0 of the clothing material and the angle &J between the incident light and the surface normal at the primary collection point are incorporated to output a reflectance response matching factor AM, which is used to redefine the reflectance characteristics of the product under the current lighting. Finally, in S4, the reflection response matching factor AM and the spatial illumination balance NG, as well as the color difference Dc and edge blur rate Nm between the image rendering result and the real image in the post-processing process, are comprehensively considered to construct a visual realism index EV, and a secondary comparative evaluation is performed with the quality threshold Eth set based on visual psychology theory to determine whether to trigger the reconstruction rendering strategy, and further perform sampling frame rate improvement, shadow enhancement or perturbation parameter fine-tuning. In summary, this embodiment achieves the purpose of intelligent response, adaptive adjustment and video realism enhancement of product materials to different lighting environments without hardware changes by establishing a closed loop from illumination acquisition to spatial adaptation to material dynamic perturbation to visual evaluation. Compared with traditional try-on solutions based on static mapping or fixed illumination models, the present invention has significant advantages in improving the consistency of illumination restoration, the authenticity of material reflection and the overall visual credibility. It can effectively solve the common problems of illumination distortion and rendering mismatch in current virtual try-on systems, and bring users a more realistic, natural and immersive virtual try-on experience.

[0064] Example 2

[0065] See also Figure 1 and Figure 3 ,Specifically: S1 includes S11, S12 and S13;

[0066] S11, setting the forehead, chest, and knees of the user trying on clothes as virtual collection points, and using a visual sensor to collect color video frames of the collection points to obtain video data of the user;

[0067] S12. directly connecting the visual sensor to the virtual model using a direct line connection, transmitting the acquired video data to the virtual model, and performing computer vision and image processing on the video data in the virtual model;

[0068] Computer vision and image processing uses the OpenCV library to load video data and extract each frame from the video data to obtain the frame image. The frame image is converted into a grayscale image. MediaPipe and OpenPose are used to detect human key points in the grayscale image. The three key points of the human body in the frame image (the forehead, chest, and knees) are identified and set as the corresponding ROI points of interest.

[0069] S13. Based on the ROI point of interest region in the grayscale image, feature extraction is performed to obtain an ambient lighting vector set, and the ambient lighting vector set is subjected to Z-Score normalization processing to eliminate the dimensionality influence of all parameters in the ambient lighting vector set, thereby obtaining a standardized data set.

[0070] The standardized data set includes the light intensity I at the i-th collection point i , the angle J between the incident light at the i-th collection point and the surface normal i and the illumination gradient difference Gz between the i-th collection point and the j-th collection point ij ;

[0071] The light intensity I at the i-th collection point i Extract the mean value of the pixel values ​​within the ROI interest area in the grayscale image;

[0072] The angle J between the incident light at the i-th collection point and the surface normal i By using the DeepLearning-based method to generate a 3D human body model, the normal vectors of all acquisition points are obtained. The light source direction is inferred from the highlight points in the frame image through the Phong reflection model to obtain the light source direction vector. The angle between the light source direction vector and the normal vectors of all acquisition points is calculated using the dot product formula. The specific calculation formula is: , where represents the light source direction vector, represents the normal vector of the i-th acquisition point, arccos represents the inverse cosine function, where the light source direction vector and the normal vector of the i-th acquisition point are dimensionless vectors;

[0073] The illumination gradient difference Gz between the i-th collection point and the j-th collection point ij Based on the light intensity I of the collection point i i and the angle J between the incident light and the surface normal at the i-th collection point i Perform gradient difference calculation to obtain the illumination gradient difference between the i-th collection point and the j-th collection point, and perform average extraction on each collection point i and all collection points. The specific calculation formula is: , where n represents the total number of collection points, I jIndicates the light intensity at the jth collection point, J j It represents the angle between the incident light and the surface normal at the jth acquisition point. a1 and a2 represent the normalized weight coefficients of the light intensity I and the angle J between the incident light and the surface normal, respectively. They are used to balance the intensity difference and direction difference and are set by the user.

[0074] In this embodiment, S1 of the method lays a data foundation for subsequent lighting evaluation and material mapping by constructing a complete image acquisition and lighting parameter extraction mechanism. Specifically, in sub-step S11, by setting the user's forehead, chest and knees as virtual acquisition points, and using the visual sensor to collect its color video frames, it is ensured that the collected information covers lighting conditions at different heights and different incident angles. In S12, by directly connecting the visual sensor to the virtual model system, the real-time and synchronization of data transmission are guaranteed. Then, the OpenCV library is used to grayscale the video frame, and the MediaPipe and OpenPose technologies are used to accurately locate the key points of the human body, so as to set the ROI point of interest area with high precision and provide structured coordinates for subsequent image analysis. In sub-step S13, brightness feature extraction, reflection modeling and three-dimensional posture recognition are performed on the point of interest area. Specifically including: estimating the light intensity I of each i-th acquisition point by the mean of the pixel values ​​of the ROI point of interest area in the frame image. i ; Use the deep learning model to generate a 3D human body mesh and obtain the normal vector of each acquisition point; then combine the highlight area in the frame image to infer the light source direction, and apply the dot product and arc cosine function to calculate the angle J between the incident light and the surface normal at the i-th acquisition point i On this basis, we further construct the illumination gradient difference Gz between the i-th collection point and the j-th collection point ij , which incorporates the differences in brightness and incident angle between acquisition points. All of the above parameters are normalized using the Z-score method to form a unified illumination vector normalized dataset, eliminating interference caused by dimensional differences.

[0075] Example 3

[0076] See also Figure 1 , specifically: S2 includes S21 and S22;

[0077] S21. Based on the standardized data set, calculate and output the spatial illumination balance NG, which measures the uniformity and consistency of illumination received by the entire user's ROI area of ​​interest;

[0078] The spatial illumination balance NG is calculated and output by the following algorithm formula;

[0079] ;

[0080] Where n represents the total number of acquisition points, and cos represents the cosine function;

[0081] Indicates the effective illumination difference between each two collection points. Taking into account the illumination projection component of the incident angle, divided by I i +I j The total brightness is normalized to avoid the absolute brightness point dominating the result due to differences in clothing material or skin color. This item measures the difference in incident effective illuminance between two points;

[0082] Introduce the illumination gradient difference Gz between the i-th collection point and the j-th collection point ij Used to analyze spatial unevenness factors, that is, to control the uneven weights between each pair of points due to spatial position, occlusion, environmental structure, etc. This can reflect different impact weights between the forehead and chest and the chest and knees for the same illumination difference;

[0083] Finally, 1-effective illumination difference × spatial unevenness factor is used to convert it into a balance score. The more uniform, the higher the score. Therefore, the overall difference value is converted into a balance index. Perform self-comparison on all collected points, that is, calculate the average of all collected points and output a whole-body illumination balance score;

[0084] The physical meaning of the formula lies in that during virtual try-on, different parts of the user's body, such as the forehead, chest, and knees, may experience obvious uneven lighting due to issues such as the direction of the light source and environmental occlusion. We must not only consider brightness, but also the incident direction of the light source, the orientation of the material surface, that is, the normal vector, and the degree of light and dark changes in the spatial distribution. The product of the effective illumination difference and the spatial unevenness factor is used to express the consistency of the whole-body lighting, and then perform mean calculations to analyze uniformity.

[0085] S22. Based on the tolerance range of perceptible differences for the human eye, set illumination uniformity interval thresholds. The illumination uniformity interval thresholds include a first illumination threshold F1 and a second illumination threshold F2. Perform a preliminary comparative evaluation of the spatial illumination balance NG obtained in real time with the illumination uniformity interval thresholds to determine the uniformity of the ambient illumination distribution for the user trying on the device. Based on the preliminary comparative evaluation results, the current user trying on the device is divided into different levels. The specific evaluation contents are as follows;

[0086] In the human visual system: changes in brightness of less than 15% due to lighting or incident direction are usually not noticeable;

[0087] A difference of 20%-30% begins to cause the human eye to intuitively react to the material as unnatural; exceeding 30% will make the try-on video clearly feel unrealistic. Therefore, the first illumination threshold F1 is set to the 85th percentile of the spatial illumination balance NG, and the second illumination threshold is set to the 70th percentile of the spatial illumination balance NG.

[0088] When the spatial illumination balance NG ≥ the first illumination threshold F1, it indicates that the ambient illumination distribution is uniform and consistent. In this case, it is classified as G1 level. The original material parameters are used without disturbance and standard rendering is performed directly.

[0089] When the second illumination threshold F2 ≤ spatial illumination balance NG < the first illumination threshold F1, it indicates that there is partial illumination deviation and local vignetting, and the lighting is classified as level G2.

[0090] When the spatial illumination balance NG is less than the second illumination threshold F2, it indicates that the ambient light has abnormal contrast, the environment is asymmetric, the incident angle is chaotic, and the ambient light is bad. In this case, it is classified as G3 level.

[0091] Summarize the G1 level, G2 level and G3 level, and output the level label LG. The specific form of the level label LG is LG∈{G1,G2,G3}.

[0092] In this embodiment, step S2 of the method establishes a spatial illumination balance calculation and evaluation mechanism, achieving a quantitative and hierarchical understanding of the illumination distribution on the user's body surface, effectively improving environmental adaptability and material matching capabilities before virtual rendering. Specifically, step S21 calculates the spatial illumination balance NG based on a standardized illumination dataset, comprehensively considering the effective illumination differences between different acquisition points to model illumination consistency for areas such as the user's forehead, chest, and knees. By multiplying the effective illumination differences between pairs of points by a gradient factor for normalization and averaging all pairwise combinations of acquisition points, the impact of environmental errors such as localized abnormal light spots and shadows on the overall evaluation is effectively suppressed, thereby outputting a spatial illumination balance NG that reflects overall illumination uniformity. In step S22, based on the human eye's perceptible range of brightness changes and with reference to the 15% and 30% thresholds of the human eye's perceptible difference tolerance range, a first illumination threshold F1 and a second illumination threshold F2 are set, corresponding to the 85th and 70th percentiles of the spatial illumination balance NG, respectively. Then, based on the numerical range of NG, the user environment is divided into three levels: G1 uniform lighting, G2 moderate disturbance, and G3 severe skewed lighting, and a level label LG is output. This label will be used to guide the subsequent material disturbance degree, reflection response adjustment, and rendering scheme selection. Through the construction and application of the above-mentioned lighting balance evaluation system, the present invention has achieved the following beneficial effects: it realizes the data-driven quantitative identification of the lighting environment before virtual try-on, without relying on additional hardware; it ensures the adaptability of the material mapping strategy, so that virtual products under different lighting levels still maintain a sense of reality; it reduces distortion phenomena such as false shadows and false highlights caused by uneven lighting, and improves the naturalness and credibility of the final try-on rendering. Overall, it effectively improves the robustness of the system and the quality of user subjective experience in multi-environment and multi-light source scenarios.

[0093] Example 4

[0094] See also Figure 1 ,Specifically: S3 includes S31 and S32;

[0095] S31. Based on the level label LG, it is converted into a dynamic disturbance control factor RD (LG), and its specific form is set as:

[0096] .

[0097] Example 2, please refer to Figure 1 Specifically: S32, based on the dynamic perturbation control factor RD (LG) combined with the basic reflectivity R0 of the clothing material and the angle &J between the incident light and the surface normal at the main collection point, calculate the output reflection response matching factor AM, and dynamically perturb the material reflection according to the environment level;

[0098] The reflection response matching factor AM is calculated and output by the following algorithm formula;

[0099] ;

[0100] In the formula, R0 automatically matches the material output of different tried-on clothes through the material database of the virtual model. Indicates the material disturbance sensitivity coefficient, which is used to control the sensitivity of the material of the current clothing being tried on to light disturbance. The specific value is set by the user. It represents the attenuation factor of the illumination effect, which is used to control the attenuation rate of the incident angle deviation on the disturbance. Its specific value is set by the user and is dimensionless. e represents the exponential function.

[0101] The basic reflectivity R0 of the clothing material represents the ideal reflective performance and serves as the reference coefficient of the disturbance;

[0102] Indicates the disturbance enhancement phase, indicating whether the disturbance is turned on and the magnitude of the disturbance. G1 level is not turned on, and G3 level is the maximum.

[0103] The entire formula is used to dynamically fine-tune the reflective performance of the material, allowing it to actively adapt to environmental changes and enhance the realism and consistency of virtual try-ons.

[0104] In this embodiment, step S3 dynamically adjusts the reflective properties of virtual clothing materials by introducing a material response control mechanism based on the environmental level label LG to adapt to varying lighting conditions, ensuring that the try-on video maintains a high level of visual fidelity in all scenarios. Specifically, in step S31, the dynamic perturbation control factor RD(LG) is mapped to the light level label LG∈{G1,G2,G3} output in the previous step. This factor numerically defines whether perturbation is enabled and its intensity. In the G1 environment, no perturbation is required; in the G2 environment, moderate perturbation is triggered; and in the G3 environment, maximum perturbation is triggered. This ensures that the material responds more proactively to adverse lighting conditions. In step S32, the dynamic perturbation control factor RD(LG) is combined with parameters such as the base reflectivity R0 of the clothing material and the angle &J between the incident light and the surface normal at the primary acquisition point, and substituted into the formula for calculating the reflective response matching factor AM. This formula enhances reflectivity in low-light conditions and automatically converges highlights to avoid overexposure or distortion. The introduction of this dynamic perturbation material adjustment mechanism has achieved the following technical effects: the material performance has light adaptability, ensuring that natural reflections and textures can still be presented in environments with severe light and dark contrasts; it compensates for the asymmetric changes in the environment that standard rendering cannot capture, so that virtual clothing has realistic visual consistency during dynamic try-on; it improves the stability of the user's try-on experience under different lighting conditions, and effectively suppresses the plastic feel and distortion caused by being too dark or too bright. Overall, the implementation of S3 further provides a highly robust and adaptable material input foundation for subsequent rendering processing, significantly improving the system's intelligent response capabilities and visual quality assurance level.

[0105] Example 5

[0106] See also Figure 1 , specifically: S4 includes S41, S42 and S43;

[0107] S41. After dynamically disturbing the material reflection, render the ROI area of ​​interest of the user trying on the clothes to obtain a video rendering frame image. Then, perform a difference calculation on the average pixel value of the ROI area of ​​interest of the video rendering frame image and the real shot frame image, and output the color difference Dc between the rendering result and the real image;

[0108] Use the Mask semantic segmentation output mask to extract the edge area of ​​the material of the clothing being tried on in the rendered frame image. Apply the Sobel gradient operator to the grayscale image of the edge area to extract the gradient amplitude. Calculate the ratio of the standard deviation of the gradient amplitude to the upper limit of the gradient amplitude to obtain the acquisition edge blur rate Nm.

[0109] The color difference Dc and edge blur rate Nm between the rendering result and the real image are normalized using Z-Score to eliminate the dimension effect;

[0110] The reflection response matching factor AM and the spatial illumination balance NG are combined with the color difference Dc and edge blur rate Nm between the rendering result and the real image to calculate the visual realism index EV, which measures the consistency between the rendering result and the real image.

[0111] The visual realism index EV is calculated and output by the following algorithm formula;

[0112] ;

[0113] Where, Indicates color and boundary deviation. The visual realism index EV has a directional influence. The larger the value, the stronger the visual distortion.

[0114] The material effect multiplied by the lighting adaptation determines the upper limit of basic realism. Color error plus unclear edges cause visual disparity, and the visual realism index EV is suppressed through the derivative structure.

[0115] S42. Based on the visual psychology JND model, a visual quality threshold Eth is set. The visual realism index EV obtained in real time is subjected to a secondary comparative evaluation to determine the eligibility of the current rendering of the user trying on clothes. The specific evaluation contents are as follows;

[0116] When the visual realism index EV ≥ the visual quality threshold Eth, it means rendering and,maintaining the current strategy;

[0117] When the visual realism index EV is less than the visual quality threshold Eth, it means that the rendering is unqualified, and the intelligent intervention strategy is activated;

[0118] Among them, the JND model theory of visual psychology points out that when the change in visual quality indicators is lower than the threshold of human eye perception, about 5-10%, users cannot distinguish the difference in the picture (such as Figure 1 The setting of the visual quality threshold Eth means that it is qualified when it is less than 5% lower than the maximum value of the theoretical visual realism index EV, ensuring a good subjective experience for users;

[0119] Figure 1 :Visual Psychology JND Model Experiment Example Table

[0120] Sample No. EV value Deviation from maximum value Difference Percentage User subjective feedback A1 1.30 0.00 0% Totally true, excellent A2 1.22 0.08 2.3% Slight color / edge difference A3 1.10 0.20 15.4% Obvious color cast / texture distortion

[0121] S43. After the rendering fails, an intelligent intervention strategy is initiated, wherein the intelligent intervention strategy includes visual sensor intelligent intervention, shadow intelligent intervention, and material disturbance intelligent intervention;

[0122] Intelligent sensor intervention increases the current sampling frame rate by 50% to obtain more key frames, improving boundary clarity and dynamic accuracy;

[0123] Intelligent shadow intervention renders virtual shadows at the highlight points in the ROI area of ​​interest to compensate for brightness differences or highlight overflow, increasing realism.

[0124] Intelligent intervention of material disturbance automatically adjusts the material disturbance sensitivity coefficient in the visual realism index EV Increase by 50%, while affecting the attenuation factor of the light Reduce by 50% to enhance the dynamic reflection of highlights and shadows depending on the angle or environment, and make up for the lack of low-light performance.

[0125] In this embodiment, the method completes the comprehensive verification and final authenticity evaluation of the previous material and lighting response adjustment effects in S4, forming a complete feedback control closed loop. S41 first compares the virtual try-on image after perturbed reflection with the real image, extracts the color difference Dc and edge blur rate Nm of the rendering results of key areas such as the forehead, chest, and knees and the real image, combines the reflection response matching factor AM and spatial lighting balance NG outputted in the previous order, and uniformly inputs them into the calculation model of the visual authenticity index EV to comprehensively measure the degree of fit between the current virtual rendering and the real picture in multiple dimensions such as color, clarity, and lighting consistency. The index adopts a normalized fusion structure, combined with Sobel edge gradient and Z-Score normalization, to eliminate the dimensional difference between brightness and boundary information, and ensure the scientificity and objectivity of the evaluation. S42 introduces the visual psychology JND model JustNoticeableDifference as an evaluation standard, sets the visual quality threshold Eth, and takes the 5% perception error of the human eye as the tolerance limit. When the visual realism index EV is higher than the threshold, it means that the rendering is credible and no intervention is required; when the visual realism index EV is lower than the threshold, it automatically enters the intelligent intervention strategy defined by S43 to comprehensively improve the system's adaptability. S43 triggers three specific intervention mechanisms based on the differences in error sources: Visual sensor intelligent intervention: By increasing the frame rate sampling by 50%, more continuous video frame data is obtained to enhance the ability to capture and restore dynamic boundaries; Shadow intelligent intervention: Automatically generate shading shadows or soften highlight maps for areas with excessive light or highlights, correct non-realistic lighting effects, and enhance material fit; Material perturbation intelligent intervention: Through the material perturbation sensitivity coefficient Increase by 50%, while affecting the attenuation factor of the light A 50% reduction allows virtual materials to present more realistic reflection changes under dynamic lighting, effectively compensating for insufficient performance of highlights and shadows, especially in low-light environments.

[0126] Example 6

[0127] See also Figure 1 and Figure 2, a real-time product try-on video generation system based on virtual modeling, including an environmental parameter acquisition and processing module, a spatial lighting balance analysis module, a grade label output module and a rendering result analysis module;

[0128] The environmental parameter acquisition and processing module collects video data of users trying on clothes through visual sensors, and transmits the video data to the virtual model for computer vision and image processing to obtain the environmental lighting vector set. The environmental lighting vector set is then preprocessed to obtain a standardized data set.

[0129] The spatial lighting balance analysis module calculates and outputs the spatial lighting balance NG based on a standardized data set, sets a lighting uniformity interval threshold, performs a preliminary comparative evaluation with the lighting balance NG, divides the lighting level, and outputs a level label LG;

[0130] The grade label output module calculates and outputs the reflection response matching factor AM of the virtual material based on the grade label LG, redefining the reflectivity of the product material;

[0131] The rendering result analysis module calculates and outputs the visual realism index EV through the reflection response matching factor AM and the spatial illumination balance NG, sets the visual quality threshold Eth and the visual realism index EV for secondary comparative evaluation, and triggers the reconstruction rendering strategy based on the evaluation results.

[0132] While the embodiments of the present invention have been shown and described, it will be apparent to those skilled in the art that various changes, modifications, substitutions, and alterations can be made to the embodiments without departing from the principles and spirit of the invention.

Claims

1. A method for generating real-time product try-on videos based on virtual modeling, characterized by: The following steps are involved: S1. Collect video data of users trying on clothes through visual sensors, and transmit the video data to the virtual model for computer vision and image processing to obtain an ambient light vector set. Then, pre-process the ambient light vector set to obtain a standardized data set. S2. Based on the standardized data set, calculate and output the spatial illumination balance NG, set the illumination uniformity interval threshold, perform preliminary comparative evaluation with the illumination balance NG, divide the illumination levels, and output the level label LG; S3. Calculate and output the reflection response matching factor AM of the virtual material based on the grade label LG, and redefine the reflection condition of the product material; S4. Calculate and output the visual realism index EV through the reflection response matching factor AM and the spatial illumination balance NG, set the visual quality threshold Eth and the visual realism index EV for secondary comparative evaluation, and trigger the reconstruction rendering strategy based on the evaluation results.

2. The method for generating real-time product try-on videos based on virtual modeling according to claim 1, characterized in that: Said S1 includes S11, S12 and S13; S11, setting the forehead, chest, and knees of the user trying on clothes as virtual collection points, and using a visual sensor to collect color video frames of the collection points to obtain video data of the user; S12. directly connecting the visual sensor to the virtual model using a direct line connection, transmitting the acquired video data to the virtual model, and performing computer vision and image processing on the video data in the virtual model; The computer vision and image processing method uses the OpenCV library to load video data, extracts each frame in the video data, obtains a frame image, converts the frame image into a grayscale image, and uses MediaPipe and OpenPose to detect human key points on the grayscale image, identifies the three human key points of the forehead, chest and knees in the frame image, and sets them as the corresponding ROI area of ​​interest; S13. Based on the ROI point of interest region in the grayscale image, feature extraction is performed to obtain an ambient lighting vector set, and the ambient lighting vector set is subjected to Z-Score normalization processing to eliminate the dimensionality influence of all parameters in the ambient lighting vector set, thereby obtaining a standardized data set. The standardized data set includes the light intensity I of the i-th collection point i , the angle J between the incident light at the i-th collection point and the surface normal i and the illumination gradient difference Gz between the i-th collection point and the j-th collection point ij ; The light intensity I of the i-th collection point i Extract the mean value of the pixel values ​​within the ROI interest area in the grayscale image; The angle J between the incident light at the i-th collection point and the surface normal i By using the DeepLearning-based method to generate a 3D human body model, the normal vectors of all collected points are obtained. The light source direction is inferred from the highlight points in the frame image through the Phong reflection model to obtain the light source direction vector. The dot product formula is used to calculate the angle between the light source direction vector and the normal vectors of all collected points. The illumination gradient difference Gz between the i-th collection point and the j-th collection point ij Based on the light intensity I of the collection point i i and the angle J between the incident light and the surface normal at the i-th collection point i Perform gradient difference calculation to obtain the illumination gradient difference between the i-th collection point and the j-th collection point, and perform average extraction on each collection point i and all collection points.

3. The method for generating real-time product try-on videos based on virtual modeling according to claim 2, characterized in that: Said S2 includes S21 and S22; S21. Based on the standardized data set, calculate and output the spatial illumination balance NG to measure the uniformity and consistency of the illumination received by the entire user's ROI area of ​​interest.

4. The method for generating real-time product try-on videos based on virtual modeling according to claim 3, characterized in that: S22. Setting illumination uniformity thresholds based on the tolerance range of perceptible differences for the human eye. The illumination uniformity thresholds include a first illumination threshold F1 and a second illumination threshold F2. Preliminary comparison and evaluation are performed between the spatial illumination balance NG obtained in real time and the illumination uniformity thresholds to determine the uniformity of the ambient illumination distribution for the user trying on the device. Based on the preliminary comparison and evaluation results, the current user trying on the device is divided into different levels. The specific evaluation contents are as follows; When the spatial illumination balance NG ≥ the first illumination threshold F1, it indicates that the ambient illumination distribution is uniform and consistent. In this case, it is classified as G1 level. The original material parameters are used without disturbance and standard rendering is performed directly. When the second illumination threshold F2 ≤ spatial illumination balance NG < the first illumination threshold F1, it indicates that there is partial illumination deviation and local vignetting, and the lighting is classified as level G2. When the spatial illumination balance NG is less than the second illumination threshold F2, it indicates that the ambient light has abnormal contrast, the environment is asymmetric, the incident angle is chaotic, and the ambient light is bad. In this case, it is classified as G3 level. The G1 level, the G2 level, and the G3 level are aggregated to output a level label LG, wherein the specific form of the level label LG is LG∈{G1,G2,G3}.

5. The method for generating real-time product try-on videos based on virtual modeling according to claim 4, characterized in that: Said S3 includes S31 and S32; S31. Based on the level label LG, it is converted into a dynamic disturbance control factor RD (LG), and its specific form is set as: 。 6. The method for generating real-time product try-on videos based on virtual modeling according to claim 5, characterized in that: S32. Based on the dynamic perturbation control factor RD (LG) combined with the basic reflectivity R0 of the clothing material and the angle &J between the incident light and the surface normal at the main collection point, the output reflection response matching factor AM is calculated, and the material reflection is dynamically perturbed according to the environmental level.

7. The method for generating real-time product try-on videos based on virtual modeling according to claim 6, characterized in that: Said S4 includes S41, S42 and S43; S41. After dynamically disturbing the material reflection, render the ROI area of ​​interest of the user trying on the clothes to obtain a video rendering frame image. Then, perform a difference calculation on the average pixel value of the ROI area of ​​interest of the video rendering frame image and the real shot frame image, and output the color difference Dc between the rendering result and the real image; Use the Mask semantic segmentation output mask to extract the edge area of ​​the material of the clothing being tried on in the rendered frame image. Apply the Sobel gradient operator to the grayscale image of the edge area to extract the gradient amplitude. Calculate the ratio of the standard deviation of the gradient amplitude to the upper limit of the gradient amplitude to obtain the acquisition edge blur rate Nm. The color difference Dc and edge blur rate Nm between the rendering result and the real image are normalized using Z-Score to eliminate the dimension effect; The reflection response matching factor AM and the spatial illumination balance NG are combined with the color difference Dc and edge blur rate Nm between the rendering result and the real image to calculate and output the visual realism index EV, which measures the consistency between the rendering result and the real picture.

8. The method for generating real-time product try-on videos based on virtual modeling according to claim 7, characterized in that: S42. Based on the visual psychology JND model, a visual quality threshold Eth is set. The visual realism index EV obtained in real time is subjected to a secondary comparative evaluation to determine the eligibility of the current rendering of the user trying on clothes. The specific evaluation contents are as follows; When the visual realism index EV ≥ the visual quality threshold Eth, it means rendering and,maintaining the current strategy; When the visual realism index EV is less than the visual quality threshold Eth, it means that the rendering is unqualified, and the intelligent intervention strategy is activated.

9. The method for generating real-time product try-on videos based on virtual modeling according to claim 8, characterized in that: S43. After the rendering fails, an intelligent intervention strategy is initiated, wherein the intelligent intervention strategy includes visual sensor intelligent intervention, shadow intelligent intervention, and material disturbance intelligent intervention; The visual sensor intelligent intervention is achieved by increasing the current sampling frame rate by 50%; The shadow intelligent intervention renders virtual shadows by highlighting points in the ROI area of ​​interest; The material disturbance intelligent intervention automatically adjusts the material disturbance sensitivity coefficient in the visual realism index EV to Increase by 50%, while affecting the attenuation factor of the light Reduced by 50%.

10. A system for generating real-time product try-on and fitting videos based on virtual modeling, applied to the method for generating real-time product try-on and fitting videos based on virtual modeling according to any one of claims 1 to 9, characterized in that: It includes environmental parameter acquisition and processing module, spatial lighting balance analysis module, grade label output module and rendering result analysis module; The environmental parameter acquisition and processing module collects video data of users trying on clothes through visual sensors, and transmits the video data to the virtual model for computer vision and image processing to obtain an environmental lighting vector set, and then pre-processes the environmental lighting vector set to obtain a standardized data set; The spatial illumination balance analysis module calculates and outputs the spatial illumination balance NG based on the standardized data set, sets the illumination uniformity interval threshold, performs preliminary comparative evaluation with the illumination balance NG, divides the illumination level, and outputs the level label LG; The grade label output module calculates and outputs the reflection response matching factor AM of the virtual material based on the grade label LG, thereby redefining the reflectivity of the commodity material. The rendering result analysis module calculates and outputs the visual realism index EV through the reflection response matching factor AM and the spatial illumination balance NG, sets the visual quality threshold Eth and the visual realism index EV for secondary comparative evaluation, and triggers the reconstruction rendering strategy based on the evaluation results.

Images