Mirror makeup interaction display method and system fused with AR technology

Abstract

This application provides a mirror-based makeup try-on interactive display method and system integrating AR technology, relating to the field of interactive display technology. It acquires real-time mirror image data and posture / motion data of the user's face, constructs a 3D facial model based on the real-time mirror image data, and drives the 3D facial model to perform synchronous movement through the posture / motion data. Responding to the user's selection command for a target makeup look, it acquires the corresponding AR makeup layer, overlays and renders the AR makeup layer onto the 3D facial model, generating an AR makeup try-on preview model. It adjusts the lighting attributes of the AR makeup try-on preview model according to ambient light parameters to generate AR makeup effect data. The AR makeup effect data is then displayed on the display surface of the semi-transparent, semi-reflective mirror display device. This application can achieve consistent integration of virtual makeup and the real face, improving the precision and interactive experience of virtual makeup try-on in mirror-based makeup try-on scenarios.

Description

Technical Field

[0001] This application relates to the field of interactive display technology, and more specifically, to a mirror-based makeup interactive display method and system that integrates AR technology. Background Technology

[0002] Currently, augmented reality virtual makeup try-on technology has been initially applied in the beauty retail and smart hardware sectors. Existing solutions typically capture user facial images through a camera, use facial landmark detection algorithms to locate facial feature areas, and overlay a pre-set AR makeup layer onto the corresponding areas of the user's facial image as a planar texture, displaying a preview of the makeup effect on a screen. Some solutions further incorporate 3D facial reconstruction technology, allowing the makeup layer to change perspective as the user's head rotates. In addition, some smart mirror products have implemented a semi-transparent, semi-reflective mirror structure at the hardware level, capable of reflecting the user's real image while displaying digital content on a built-in screen. Regarding ambient light processing, technologies exist that use light sensors to collect ambient light intensity and adjust screen brightness accordingly.

[0003] However, when AR makeup try-on is applied to semi-transparent, semi-reflective mirror devices, the mirror reflection causes the virtual facial image to flip left-right, resulting in the makeup being rendered upside down. Furthermore, existing solutions tend to cause stretching distortion in areas such as the nose and lip edges when the makeup is applied to curved surfaces. In terms of ambient lighting adaptation, they only adjust the screen brightness without ensuring the lighting response of the makeup material matches the real face, and they fail to utilize the optical properties of the semi-transparent, semi-reflective mirror to achieve true fusion between the virtual makeup and the real facial mirror image. Therefore, achieving consistent fusion between virtual makeup and the real face, and improving the precision and interactive experience of virtual makeup try-on in mirror scenarios, is a challenge facing the industry. Summary of the Invention

[0004] This application provides a mirror makeup trial interactive display method and system that integrates AR technology, which can achieve consistent integration between virtual makeup and real face, and improve the precision and interactive experience of virtual makeup trial in mirror makeup trial scenarios.

[0005] In a first aspect, this application provides a mirror-based makeup trial interactive display method integrating AR technology, the interactive display method comprising the following steps: The system acquires real-time mirror data and posture data of the user's face in the reflective surface of a semi-transparent and semi-reflective display device, constructs a three-dimensional facial model based on the real-time mirror data, and drives the three-dimensional facial model to perform synchronous movement through the posture data. In response to the user's selection command for a target makeup in the preset makeup library, the corresponding AR makeup layer is obtained, and the AR makeup layer is overlaid and rendered onto the three-dimensional facial model to generate an AR makeup preview model. Obtain the current ambient light parameters, adjust the lighting attributes of the AR makeup preview model according to the ambient light parameters, and generate AR makeup effect data; The AR makeup effect data is displayed on the display surface of the semi-transparent, semi-reflective mirror display device.

[0006] In this embodiment, the real-time mirror data includes: facial contour data, texture data, and pose data of the user's face in the reflective surface of the semi-transparent and semi-reflective mirror display device, as well as multi-angle reflection imaging data from image acquisition units in different orientations; The posture and motion data includes: the user's head position parameters and facial expression parameters. The head position parameters include head rotation angle, pitch angle, and lateral tilt angle. The facial expression parameters include the relative displacement changes between facial key points.

[0007] In this embodiment, constructing a three-dimensional facial model based on the real-time mirror data and driving the three-dimensional facial model to perform synchronous movement through the posture and motion data specifically includes: Two-dimensional feature point set and multi-angle reflection imaging data of the user's face are extracted from the real-time mirror data. The two-dimensional feature point set and the multi-angle reflection imaging data are stereo matched and mapped to three-dimensional space to generate an initial three-dimensional facial mesh model. Head pose parameters are extracted from the posture and motion data. A rotation transformation is applied to the initial three-dimensional facial mesh model based on the head pose parameters so that the overall orientation of the three-dimensional facial model is consistent with the user's real-time head posture. Facial expression parameters are extracted from the posture and motion data, and the vertices of the corresponding regions in the three-dimensional facial mesh model are driven to shift along the normal direction based on the facial expression parameters, so that the local deformation of the three-dimensional facial model is synchronized with the user's real-time facial expression.

[0008] In this embodiment, overlaying and rendering the AR makeup layer onto the 3D facial model to generate an AR makeup preview model specifically includes: The AR makeup layer is transformed from the image coordinate system to the local surface coordinate system of the 3D facial model through inverse projection mapping, so that the AR makeup layer fits the irregular surface shape of the 3D facial model. The compensation coefficient matrix is ​​determined based on the local surface coordinate system of the three-dimensional facial model at the corresponding location, and the compensation coefficient matrix is ​​used to apply a coordinate offset that is negatively correlated with the curvature to each pixel coordinate of the AR makeup layer; The compensated AR makeup layer is then pixel-wise fused with the corresponding parts of the 3D facial model to generate an AR makeup preview model.

[0009] In this embodiment, the AR makeup preview model is a three-dimensional digital facial model used to characterize the virtual makeup effect of the user under the target makeup.

[0010] In this embodiment, adjusting the lighting attributes of the AR makeup preview model according to the ambient light parameters to generate AR makeup effect data specifically includes: The diffuse reflection intensity and specular reflection intensity of the makeup material in the AR makeup preview model are adjusted according to the light intensity and color temperature in the ambient light parameters. The AR makeup preview model's surface highlights and projection orientation are remapped based on the lighting direction in the ambient light parameters to generate AR makeup effect data. The ambient light parameters are used as real-time shading inputs in the rendering pipeline of the AR makeup preview model for pixel-by-pixel lighting calculation.

[0011] In this embodiment, adjusting the diffuse reflection intensity and specular reflection intensity of the makeup material in the AR makeup preview model according to the light intensity and color temperature in the ambient light parameters specifically includes: The hue offset vector of the current ambient light is determined based on the color temperature, and the original hue value of the AR makeup layer is offset by the hue offset vector to generate an environment-adaptive hue value. The diffuse reflection gain coefficient and the specular reflection gain coefficient are determined based on the light intensity. The diffuse channel intensity of the AR makeup layer is modulated using the diffuse gain coefficient; The specular reflection channel intensity of the AR makeup layer is modulated using the specular reflection gain coefficient.

[0012] Secondly, this application provides a mirror makeup trial interactive display system integrating AR technology, used to execute a mirror makeup trial interactive display method integrating AR technology, the interactive display system comprising: The data acquisition module is used to acquire real-time mirror data and posture and motion data of the user's face in the reflective surface of the semi-transparent and semi-reflective display device, construct a three-dimensional facial model based on the real-time mirror data, and drive the three-dimensional facial model to perform synchronous movement through the posture and motion data; The model building module is used to respond to the user's selection command for the target makeup in the preset makeup library, obtain the corresponding AR makeup layer, overlay and render the AR makeup layer onto the three-dimensional facial model, and generate an AR makeup preview model. The effect generation module is used to acquire the current ambient light parameters, adjust the lighting attributes of the AR makeup preview model according to the ambient light parameters, and generate AR makeup effect data. The effect display module is used to display the AR makeup effect data on the display surface of the semi-transparent and semi-reflective mirror display device.

[0013] Thirdly, this application provides a computer device, the computer device including a memory and a processor, the memory for storing a computer program, and the processor for calling and running the computer program from the memory, so that the computer device executes the above-described mirror makeup interactive display method integrating AR technology.

[0014] Fourthly, this application provides a computer-readable storage medium storing instructions or code that, when executed on a computer, cause the computer to implement the aforementioned mirror makeup interactive display method integrating AR technology.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The system acquires real-time mirror image data and posture / motion data of the user's face on the reflective surface of a semi-transparent, semi-reflective display device. Based on the real-time mirror image data, a 3D facial model is constructed, and the posture / motion data drives the 3D facial model to perform synchronous movement. Responding to the user's selection command for a target makeup look from a preset makeup library, the system acquires the corresponding AR makeup layer, overlays and renders the AR makeup layer onto the 3D facial model, generating an AR makeup preview model. It acquires the current ambient light parameters, adjusts the lighting attributes of the AR makeup preview model according to the ambient light parameters, and generates AR makeup effect data. The AR makeup effect data is then displayed on the display surface of the semi-transparent, semi-reflective display device.

[0016] Therefore, this application firstly acquires real-time mirror image data of the user's face through the reflective surface of a semi-transparent, semi-reflective display device, constructs a three-dimensional facial model using multi-angle image acquisition and stereo matching technology, and simultaneously drives the model to achieve synchronized movement of overall orientation and local expressions using posture and motion data, providing users with a dynamic experience foundation for free mirroring and improving the naturalness and real-time responsiveness of the virtual makeup interaction; secondly, by attaching a two-dimensional AR makeup layer to the irregular curved surface of the three-dimensional facial model, and performing conformal stretching and compression compensation on the makeup layer according to the curvature distribution, the AR makeup presents a uniform and natural coverage effect in different areas of the face, improving the precision and realism of the virtual makeup; thirdly, by dynamically adjusting the lighting properties of the makeup material through the current ambient light parameters, the lighting response of the AR makeup is kept consistent with the lighting performance of the real facial mirror under the same ambient light, improving the naturalness of the virtual-real fusion and scene adaptability; finally, the AR makeup effect data is displayed on the display surface, allowing users to see the AR makeup effect superimposed on the corresponding position while observing their own real facial mirror, improving the intuitiveness and interactive experience of the makeup.

[0017] In summary, the technical solution adopted in this application can achieve a consistent integration between virtual makeup and real face, improving the precision and interactive experience of virtual makeup try-on in mirror try-on scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this embodiment of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an exemplary flowchart of a mirror-based makeup trial interactive display method that integrates AR technology, according to the present application. Figure 2 This is a schematic diagram of a device scene for a mirror-based makeup interactive display system that integrates AR technology, according to the present application. Figure 3 This is a module structure diagram of a mirror makeup interactive display system that integrates AR technology, provided in this application.

[0020] Figure 4 This is a schematic diagram of the structure of a computer device that implements the interactive display method according to the present application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0022] This application provides a mirror-based makeup try-on interactive display method and system that integrates AR technology. The core of the method involves acquiring real-time mirror image data and posture / motion data of the user's face in the reflective surface of a semi-transparent, semi-reflective display device; constructing a three-dimensional facial model based on the real-time mirror image data; and driving the three-dimensional facial model to perform synchronous movement using the posture / motion data. In response to the user's selection command for a target makeup look from a preset makeup library, the method acquires the corresponding AR makeup layer, overlays and renders the AR makeup layer onto the three-dimensional facial model, generating an AR makeup try-on preview model. It also acquires the current ambient light parameters, adjusts the lighting attributes of the AR makeup try-on preview model according to the ambient light parameters, and generates AR makeup try-on effect data. Finally, it displays the AR makeup try-on effect data on the display surface of the semi-transparent, semi-reflective display device.

[0023] Example 1: To better understand the above technical solution, the following will provide a detailed description of the technical solution in conjunction with the accompanying drawings and specific implementation methods. (Refer to...) Figure 1 As shown in the figure, this is an exemplary flowchart of a mirror makeup trial interactive display method integrating AR technology according to this embodiment of the application. The interactive display method includes the following steps: In step S1, real-time mirror image data and posture motion data of the user's face in the reflective surface of the semi-transparent and semi-reflective display device are acquired. A three-dimensional facial model is constructed based on the real-time mirror image data, and the three-dimensional facial model is driven to perform synchronous movement through the posture motion data.

[0024] In practical implementation, real-time mirror image data and posture / motion data of the user's face in the reflective surface of the semi-transparent, semi-reflective display device are acquired. Specifically, the user stands or sits in front of the device, facing its reflective surface. Multiple image acquisition units are pre-set around the device's frame, including a first image acquisition unit located at the center of the top of the frame, and second and third image acquisition units located at the upper left and upper right corners of the frame. The optical axes of each image acquisition unit are all oriented towards the area where the user's face is located, and the fields of view of each acquisition unit overlap, collectively covering the possible movement space of the user's face. The first image acquisition unit synchronously acquires the user's facial image from a first-person perspective, serving as the main-view image in the real-time mirror data. The second and third image acquisition units synchronously acquire the user's facial images from the left and right-person perspectives, respectively, serving as auxiliary positioning data. The three image acquisition units mentioned above have undergone intrinsic and extrinsic parameter calibration at the factory. Intrinsic parameter calibration includes the focal length, principal point coordinates, and distortion coefficients of each acquisition unit. Extrinsic parameter calibration includes the relative positional relationships between the acquisition units and their spatial pose relative to the reflective surface of the semi-transparent mirror display device. Simultaneously, the image acquisition units extract a set of two-dimensional feature points of the user's face from the image sequences captured by the units. These feature points include, but are not limited to, key feature points such as the corners of the eyes, the tip of the nose, the corners of the mouth, and the jawline. The coordinate changes of these feature points in consecutive image frames are continuously tracked. Based on the coordinate changes of these feature points, the rotation angle, pitch angle, and tilt angle of the user's head in three-dimensional space are calculated as head pose parameters. Simultaneously, based on the relative displacement changes between key facial points, facial expression parameters are calculated, including but not limited to the degree of eye opening and the degree of mouth corner raising. These head pose parameters and facial expression parameters together constitute the posture and motion data. Real-time mirror data includes: facial contour data, texture data, and pose data of the user's face in the reflective surface of the semi-transparent and semi-reflective display device, as well as multi-angle reflection imaging data from image acquisition units in different orientations; posture and motion data includes: the user's head position pose parameters and facial expression parameters, wherein the head position pose parameters include head rotation angle, pitch angle, and lateral tilt angle, and the facial expression parameters include the relative displacement changes between facial key points.

[0025] It should be noted that the semi-transparent and semi-reflective mirror display device includes a reflective layer and a transmissive display layer that are bonded together. The surface of the reflective layer facing the user is the reflective surface, which is composed of a semi-transparent and semi-reflective coating. Its optical characteristics are: it has the ability to partially reflect incident light from the direction of the user's face, allowing the user to observe their own facial image on this reflective surface; at the same time, the reflective layer has the ability to partially transmit light emitted from the side of the transmissive display layer facing away from the user, allowing the light of the AR makeup effect displayed on the display surface to pass through the reflective layer and reach the user's eyes.

[0026] In this embodiment, a three-dimensional facial model is constructed based on the real-time mirror data, and the three-dimensional facial model is driven to perform synchronous movement through the posture and motion data. Specifically, this can be achieved through the following steps: Two-dimensional feature point set and multi-angle reflection imaging data of the user's face are extracted from the real-time mirror data. The two-dimensional feature point set and the multi-angle reflection imaging data are stereo matched and mapped to three-dimensional space to generate an initial three-dimensional facial mesh model. Head pose parameters are extracted from the posture and motion data. A rotation transformation is applied to the initial three-dimensional facial mesh model based on the head pose parameters so that the overall orientation of the three-dimensional facial model is consistent with the user's real-time head posture. Facial expression parameters are extracted from the posture and motion data, and the vertices of the corresponding regions in the three-dimensional facial mesh model are driven to shift along the normal direction based on the facial expression parameters, so that the local deformation of the three-dimensional facial model is synchronized with the user's real-time facial expression.

[0027] In practical implementation, firstly, a two-dimensional feature point set and multi-angle reflection imaging data of the user's face are extracted from the real-time mirror data. The two-dimensional feature point set and multi-angle reflection imaging data are then stereo-matched and mapped into three-dimensional space to generate an initial three-dimensional facial mesh model. Specifically, using the image captured by the main viewpoint image acquisition unit as a reference, an active appearance model algorithm is used to locate feature points in the facial region of the image, outputting the two-dimensional pixel coordinates of a total of sixty-eight key facial feature points, including the corners of the eyes, the tip of the nose, the corners of the mouth, and the jawline. Multi-angle reflection imaging data is extracted from the real-time mirror data, originating from the left-side and right-side viewpoint images captured by the auxiliary image acquisition units at the upper left and upper right corners of the frame. In these two-side viewpoint images, the same active appearance model algorithm is used to locate the corresponding facial feature points in the main viewpoint image. The two-dimensional feature point set in the main view image is stereo matched with the corresponding feature points in the left and right view images. Using a feature point in the main view image as the center, a pixel window of a preset size around that point is taken as the matching primitive. The window with the most similar grayscale distribution is searched along the epipolar direction corresponding to that feature point in the left and right view images; the center of this window is the location of the corresponding feature point. Once the two-dimensional coordinates of the same feature point in the main view, left view, and right view are determined, the depth value of the feature point in three-dimensional space is calculated using triangulation based on the pre-calibrated extrinsic parameter matrices between the three image acquisition units, including the relative rotation matrix and translation vector, thus obtaining the spatial three-dimensional coordinates of the feature point. All sixty-eight key facial feature points are traversed, and their spatial three-dimensional coordinates are calculated one by one to generate a sparse three-dimensional feature point cloud. Delaunay triangulation is performed on this sparse three-dimensional feature point cloud, constructing triangular patches between adjacent feature points to form an initial three-dimensional facial mesh model.

[0028] In addition, in the specific implementation, head pose parameters are extracted from the posture and motion data. A rotation transformation is then applied to the initial 3D facial mesh model based on these parameters, ensuring that the overall orientation of the 3D facial model matches the user's real-time head posture. Specifically, the head pose parameters at the current moment are extracted from the posture and motion data, including the user's head rotation angle, pitch angle, and tilt angle in 3D space. These three angles are calculated through the coordinate changes of facial feature points in consecutive frames of images. Specifically, the image coordinate changes of the nose tip feature point represent the horizontal rotation and pitch movements of the head, while the change in the angle between the line connecting the centers of the eyes and the horizontal direction represents the tilt movement of the head. These rotation angles, pitch angle, and tilt angle are then converted into rotation matrices in 3D space. The rotation matrix is ​​a 3x3 orthogonal matrix, satisfying that the transpose of the rotation matrix is ​​equal to its inverse, and the determinant is equal to one. This ensures that the rotation transformation only changes the orientation of the model's vertices without altering the model's geometry and size. The rotation matrix is ​​applied to the coordinates of all vertices of the initial 3D facial mesh model, i.e., the rotation matrix is ​​multiplied to the left of each vertex coordinate, causing the overall orientation of the 3D facial mesh model to rotate synchronously with the user's real-time head posture. Finally, facial expression parameters are extracted from the posture and motion data. Based on the facial expression parameters, the vertices of the corresponding regions in the 3D facial mesh model are driven to shift along the normal direction, so that the local deformation of the 3D facial model is synchronized with the user's real-time facial expression. That is, the facial expression parameters at the current moment are extracted from the posture and motion data. The facial expression parameters include the displacement changes of each key point of the face relative to the neutral expression state. In this embodiment, the spatial coordinates of each key point of the face in the user's neutral expression state are pre-collected as reference coordinates. After repositioning the spatial coordinates of each key point in the current frame, the displacement increment of each key point relative to the reference coordinates in the X, Y, and Z directions is calculated to form a set of displacement change vectors. Based on the aforementioned set of displacement change vectors, the vertex regions in the 3D facial mesh model corresponding to the expression changes are driven to shift along the normal direction. Specifically, when key points on both sides of the mouth are detected to shift laterally outward and vertically upward relative to the reference position, it is determined that the user is performing a smiling expression. At this time, the mesh vertices in the mouth region of the 3D facial mesh model are shifted in the same direction and magnitude. Simultaneously, the mesh vertices in the cheekbone region are slightly raised outward along the facial surface normal direction to simulate the protrusion of the cheekbone muscles when smiling. When key points on the upper and lower eyelids are detected to move away from each other vertically relative to the reference position, it is determined that the user is performing an open-eye expression. At this time, the mesh vertices in the upper and lower eyelid regions of the 3D facial mesh model are moved away from each other along the surface normal direction of their respective positions. When key points in the eyebrow region are detected to shift vertically upward relative to the reference position and key points in the glabella region shift vertically downward, it is determined that the user is performing a raised-eyebrow expression. At this time, the mesh vertices in the brow arch region of the 3D facial mesh model are raised upward along the surface normal direction.The aforementioned local deformation driving process is executed at every frame of image acquisition, and the driving frequency is consistent with the frame rate of the image acquisition unit, ensuring that the local deformation of the 3D facial model is synchronized with the user's real-time facial expressions frame by frame.

[0029] In step S2, in response to the user's selection instruction for a target makeup in the preset makeup library, the corresponding AR makeup layer is obtained, and the AR makeup layer is superimposed and rendered onto the three-dimensional facial model to generate an AR makeup preview model.

[0030] In practical implementation, in response to the user's selection command of a target makeup look from the preset makeup look library, the corresponding AR makeup layer is obtained. That is, the preset makeup look library is stored on the cloud server of the semi-transparent, semi-reflective display device. The makeup look library contains multiple makeup templates, each corresponding to a complete makeup application scheme. Each template consists of at least one makeup layer, and each layer corresponds to a specific makeup area and step, including: foundation layer, concealer layer, blush layer, contour layer, highlighter layer, eyeshadow layer, eyeliner layer, eyelash layer, eyebrow layer, and lip makeup layer. An AR makeup layer is essentially a digital image containing color and transparency information, stored as a pixel matrix. Each pixel has three color channels (RGB) and one alpha transparency channel value. The alpha transparency channel value defines the coverage intensity of that pixel in the makeup area. During the mirror-viewing process, users browse various makeup templates in the preset makeup library through the graphical user interface displayed on the semi-transparent, semi-reflective mirror display device. When a user browses to a particular makeup template, the device displays a thumbnail preview of the makeup effect on the current display surface. Users can select a target makeup look by touching the display surface of the semi-transparent, semi-reflective mirror display device. After receiving the selection command, the device parses the identifier of the target makeup template contained in the command. Based on this identifier, the device retrieves the corresponding makeup template data package from the preset makeup library. In addition to the pixel matrix data of each AR makeup layer, the makeup template data package also contains a target makeup area label corresponding to that layer, indicating which part of the 3D facial model the layer should be overlaid on.

[0031] In this embodiment, the AR makeup layer is overlaid and rendered onto the 3D facial model to generate an AR makeup preview model. This can be achieved through the following steps: The AR makeup layer is transformed from the image coordinate system to the local surface coordinate system of the 3D facial model through inverse projection mapping, so that the AR makeup layer fits the irregular surface shape of the 3D facial model. The compensation coefficient matrix is ​​determined based on the local surface coordinate system of the three-dimensional facial model at the corresponding location, and the compensation coefficient matrix is ​​used to apply a coordinate offset that is negatively correlated with the curvature to each pixel coordinate of the AR makeup layer; The compensated AR makeup layer is then pixel-wise fused with the corresponding parts of the 3D facial model to generate an AR makeup preview model.

[0032] In practical implementation, firstly, the AR makeup layer can be transformed from the image coordinate system to the local surface coordinate system of the 3D facial model through inverse projection mapping, so that the AR makeup layer fits the irregular surface shape of the 3D facial model. That is, the spatial position of the target makeup area corresponding to each pixel in the AR makeup layer on the 3D facial model is determined. Using the ray casting method, rays are emitted from each vertex of the 3D facial model in the opposite direction of the surface normal at that vertex. The pixel position that the ray passes through when it passes through the image plane of the display surface is the pixel coordinate of that vertex in the AR makeup layer. In this way, a one-to-one correspondence between each vertex of the 3D facial model and each pixel in the AR makeup layer is established. In the inverse projection mapping process, for areas in the 3D facial model where the vertex density is insufficient to accurately map makeup texture details, bilinear interpolation is performed on the mesh patches between adjacent vertices to generate virtual pixel mapping points within these patches. This ensures that all pixels of the AR makeup layer are mapped to their corresponding positions on the surface of the 3D facial model. Then, a compensation coefficient matrix is ​​determined based on the local surface coordinate system of the 3D facial model at the corresponding location. This compensation coefficient matrix applies a coordinate offset negatively correlated with the curvature to the pixel coordinates of the AR makeup layer. Specifically, for each mesh vertex covered by the target makeup area in the 3D facial model, the principal curvature values ​​along the horizontal and vertical directions are calculated. The principal curvature values ​​reflect the degree of curvature of the surface where the vertex is located in a specific direction. The average of the horizontal and vertical principal curvatures is taken as the average curvature of the vertex, and the arithmetic mean of the average curvatures of all vertices in the target makeup area is taken as the reference curvature value for that area. For convex areas with curvature greater than the reference curvature value, a first ratio between its curvature and the reference curvature is calculated, and the stretching compensation coefficient for that area is determined based on this first ratio. For concave regions with curvature less than the reference curvature value, a second ratio between their curvature and the reference curvature is calculated, and the compression compensation coefficient for that region is determined based on this second ratio. The stretching compensation coefficients and compression compensation coefficients for each region are arranged in matrix form according to the spatial positions of each vertex in the 3D facial model, thus forming the compensation coefficient matrix. When determining the compression compensation coefficient based on the second ratio, a compensation intensity factor is preset. This factor is pre-selected based on the skin elasticity characteristics of the target makeup area and can be preset through experimental data analysis. The difference between 1 and the second ratio is multiplied by the compensation intensity factor, and the resulting product is added to 1. The result is used as the compression compensation coefficient, which is a value used to quantify the degree of compression distortion in concave regions and guide the outward expansion of pixels.It should be noted that the values ​​of each element in the compensation coefficient matrix are negatively correlated with the curvature of its location. In convex regions with greater curvature, the compensation coefficient value is larger, used to counteract the overstretching effect in that region, restoring the makeup pattern to its original proportions. In concave regions with less curvature, the compensation coefficient value is smaller, used to counteract the overcompression effect in that region, restoring the makeup pattern to its original density. After obtaining the compensation coefficient matrix, coordinate offsets are applied to the pixel coordinates of the AR makeup layer using the compensation coefficient matrix. That is, for a given pixel, the horizontal coordinate value is multiplied by the corresponding compensation coefficient value, and the vertical coordinate value is also multiplied by the compensation coefficient value, causing the pixel to move outward or inward simultaneously in both the horizontal and vertical directions.

[0033] Furthermore, in the specific implementation, the compensated AR makeup layer is pixel-level fused with the corresponding parts of the 3D facial model to generate an AR makeup preview model. Specifically, for each grid vertex covered by the target makeup area of ​​the 3D facial model, the makeup pixel value corresponding to that vertex is read from the compensated AR makeup layer through the established vertex-pixel mapping relationship. The makeup pixel value includes RGB color channel values ​​and Alpha transparency channel values. Then, the user's original facial texture pixel value at the corresponding vertex position is extracted from real-time mirror data. The makeup color value of this pixel is multiplied by its Alpha value to obtain the makeup contribution component. The original texture color value is multiplied by the difference between the original texture color value and the Alpha value to obtain the texture contribution component. The makeup contribution component and the texture contribution component are then added together, and the sum is used as the final color value of the fused pixel. During the above fusion process, for the boundary positions of the target makeup area, the Alpha value is gradually reduced from its maximum value at the boundary to zero towards the outside of the boundary to achieve a smooth transition between the makeup area and the unmade-up area. The decreasing gradient of the Alpha value is radially uniformly distributed on the curved surface of the 3D facial model. When there are multiple target makeup areas, the above blending process is performed sequentially from bottom to top. The foundation layer, as the bottom layer, is blended with the 3D facial model first. The blush and highlight layers are layered on top of the foundation layer as intermediate layers, and the lip makeup layer is layered on top of the lip makeup layer as the outermost layer. Each layer is blended pixel-by-pixel based on its respective alpha channel value, ultimately generating a complete AR makeup preview model. The AR makeup preview model is a 3D digital facial model that moves synchronously with the 3D facial model. It is used to represent the virtual makeup effect on the user under the target makeup.

[0034] In step S3, the current ambient light parameters are obtained, and the lighting attributes of the AR makeup preview model are adjusted according to the ambient light parameters to generate AR makeup effect data.

[0035] In practical implementation, the current ambient light parameters are acquired. Specifically, a first light sensor and a second light sensor are respectively positioned on the side of the display surface and the side of the reflective surface of the semi-transparent, semi-reflective mirror display device. The first light sensor is located at the center of the upper bezel of the display surface, with its photosensitive surface facing outwards, and is used to collect ambient incident light from the display surface side. The second light sensor is located on the side bezel of the reflective surface, with its photosensitive surface facing outwards, and is used to collect reflected ambient light from the reflective surface side. During the user's reflection, the first and second light sensors work synchronously, each measuring the ambient light at a preset sampling period. Within each sampling period, the sensor outputs three sets of electrical signals, corresponding to the light intensity values ​​of the red, green, and blue channels, respectively. These electrical signals are then converted from analog to digital to obtain digital illuminance values ​​and color temperature. The direction of illumination is determined by analyzing the difference in measurements between the first and second photosensors. When there is a significant difference in the light intensity values ​​collected by the two sensors, it indicates that the light mainly comes from one side; the larger the difference, the more the light direction is biased towards that side. When the light intensity values ​​collected by the two sensors are basically equal, it indicates that the ambient light is uniformly scattered light, and there is no clear dominant illumination direction. The data from these three dimensions—light intensity, color temperature, and light direction—together constitute the ambient light parameters.

[0036] In this embodiment, the lighting attributes of the AR makeup preview model are adjusted according to the ambient light parameters to generate AR makeup effect data. This can be achieved through the following steps: The diffuse reflection intensity and specular reflection intensity of the makeup material in the AR makeup preview model are adjusted according to the light intensity and color temperature in the ambient light parameters. The AR makeup preview model's surface highlights and projection orientation are remapped based on the lighting direction in the ambient light parameters to generate AR makeup effect data. The ambient light parameters are used as real-time shading inputs in the rendering pipeline of the AR makeup preview model for pixel-by-pixel lighting calculation.

[0037] In practical implementation, firstly, the diffuse and specular reflection intensities of the makeup material in the AR makeup preview model are adjusted based on the light intensity and color temperature in the ambient light parameters. This process will be explained in detail in subsequent steps. Then, the position of the highlight point and the projection orientation on the surface of the AR makeup preview model are remapped based on the light direction in the ambient light parameters, thereby generating AR makeup effect data. Specifically, the main light direction of the current environment is extracted from the ambient light parameters, and a virtual light source is constructed in the 3D scene based on the main light direction, using the light direction as the incident direction of this virtual light source. Based on the incident direction of the virtual light source, the surface of the AR makeup preview model is irradiated using a bidirectional reflectance distribution function model. This model comprehensively considers both diffuse and specular reflection components. In the calculation of the specular reflection component, the exit direction of the specular reflection light is calculated based on the incident direction of the virtual light source and the surface normal direction at each vertex of the 3D facial model, thus determining the position of the highlight point on the surface of the 3D facial model. The position of the highlight point moves with the direction of illumination. When the light shines from the left, the highlight point is located on the left side of the face; when the light shines from the right, the highlight point moves to the right side of the face. The intensity of the highlight is modulated by the specular reflection gain coefficient, and the diffusion range of the highlight is controlled by a preset surface roughness parameter. Based on the incident direction of the virtual light source, the projection area is calculated on the backlit side of the 3D facial model, generating the corresponding projection orientation. That is, at each vertex of the 3D facial model, the angle between the surface normal direction of the vertex and the incident direction of the virtual light source is compared. When the angle is greater than 90 degrees, the vertex is determined to be on the backlit side, and a shadow is applied to the vertex; when the angle is less than 90 degrees, the vertex is determined to be on the front side, and no shadow is applied. In this way, the position of the highlight point and the projection orientation on the surface of the AR makeup preview model are consistent with the highlight and shadow distribution of the real face under the same ambient lighting. The pixel color value matrix output after all the above processing is the AR makeup effect data that matches the current ambient lighting conditions. The spatial resolution of the AR makeup effect data is the physical resolution of the display surface, and the total number of pixels corresponds one-to-one with the pixel array of the display surface. One frame of AR makeup effect data is output in each frame image acquisition cycle.

[0038] It should be noted that all calculations in the above lighting attribute adjustment process are performed within the rendering pipeline of the AR makeup preview model. Ambient light parameters are used as real-time shading inputs in the pixel-by-pixel lighting calculations. That is, in the pixel shader stage of the rendering pipeline, for each pixel of the 3D facial model, the ambient light parameters of the current frame are read, and the diffuse intensity, specular intensity, and shadow value of the pixel are calculated based on these parameters. Then, the final color value of the pixel is output. Since the rendering pipeline performs the above pixel-by-pixel lighting calculations at a frequency of one frame every one-sixtieth of a second, changes in ambient light parameters can be immediately reflected in the AR makeup effect data of the display surface during the rendering of the next frame, thereby achieving real-time rendering with adaptive ambient light.

[0039] In this embodiment, the diffuse reflection intensity and specular reflection intensity of the makeup material in the AR makeup preview model are adjusted according to the light intensity and color temperature in the ambient light parameters. This can be achieved through the following steps: The hue offset vector of the current ambient light is determined based on the color temperature, and the original hue value of the AR makeup layer is offset by the hue offset vector to generate an environment-adaptive hue value. The diffuse reflection gain coefficient and the specular reflection gain coefficient are determined based on the light intensity. The diffuse channel intensity of the AR makeup layer is modulated using the diffuse gain coefficient; The specular reflection channel intensity of the AR makeup layer is modulated using the specular reflection gain coefficient.

[0040] In practical implementation, firstly, the hue shift vector of the current ambient light is determined based on the color temperature. This hue shift vector is then used to offset the original hue value of the AR makeup layer, generating an environment-adaptive hue value. In other words, the hue shift vector is a three-dimensional vector in the RGB color space representing the direction and magnitude of the color shift. When the color temperature is below 3500 Kelvin, the current ambient light is considered warm-toned. In this case, the hue shift vector points towards orange-yellow, with positive red channel components and negative blue channel components, indicating that the makeup color needs to shift towards a warmer tone. When the color temperature is above 5500 Kelvin, the current ambient light is considered cool-toned. In this case, the hue shift vector points towards blue-violet, with negative red channel components and positive blue channel components, indicating that the makeup color needs to shift towards a cooler tone. When the color temperature is between 3500 and 5500 Kelvin, the current ambient light is considered close to a neutral color temperature. In this case, all components of the hue shift vector are close to zero, indicating that the makeup color does not require hue shifting. After determining the hue offset vector, a vector addition operation is performed between this hue offset vector and the original RGB values ​​of each pixel in the AR makeup layer. That is, the corresponding offset is applied to the red, green, and blue channels of each pixel to generate an ambient adaptive hue value after adaptive adjustment of ambient color temperature. Next, the diffuse reflection gain coefficient and specular reflection gain coefficient are determined based on the light intensity. That is, for the diffuse reflection gain coefficient, the light intensity value is mapped to the diffuse reflection gain coefficient in a positive linear mapping. That is, the higher the light intensity value, the larger the diffuse reflection gain coefficient, and the lower the light intensity value, the smaller the diffuse reflection gain coefficient. When the ambient light is dark, the diffuse reflection gain coefficient is less than one, and when the ambient light is bright, the diffuse reflection gain coefficient is greater than one. This increases the intensity of the diffuse reflection channel of the makeup material to maintain the visibility of the makeup in bright environments. The diffuse reflection gain coefficient is limited to the range of 0.3 to 1.8 to prevent the makeup material from being too dark or overexposed. For the specular reflection gain coefficient, the light intensity value is mapped to the specular reflection gain coefficient using a non-linear mapping. The specular reflection gain coefficient has a non-linear positive correlation with the light intensity, and its growth rate decreases as the light intensity increases. That is, in the low light intensity range, the specular reflection gain coefficient increases rapidly with the increase of light intensity to enhance the glossiness of the makeup material in dark environments. In the high light intensity range, the growth of the specular reflection gain coefficient gradually flattens out to prevent excessive specular reflection glare in strong light environments, and the specular reflection gain coefficient is limited to the range of 0.2 to 1.5. The diffuse reflection gain coefficient refers to the value used to quantify the adjustment factor of the AR makeup material's response to the diffuse reflection component of ambient light. The specular reflection gain coefficient refers to the value used to quantify the adjustment factor of the AR makeup material's response to the specular reflection component of ambient light.

[0041] It should be noted that, in this embodiment, the mapping method for the diffuse reflection gain coefficient is as follows: An illuminance of 150 lux is used as the standard reference illuminance. Under this illuminance condition, the diffuse reflection gain coefficient is set to one, meaning no gain modulation is performed. When the ambient illuminance is below 150 lux, the diffuse reflection gain coefficient decreases linearly with decreasing illuminance, ranging from 0.3 to 1. That is, a minimum gain coefficient of 0.3 corresponds to an illuminance of 0 lux, and a gain coefficient of 1 corresponds to 150 lux. Intermediate illuminance values ​​are calculated using linear interpolation. When the ambient illuminance is above 150 lux, the diffuse reflection gain coefficient increases linearly with increasing illuminance, ranging from 1 to 1.8. That is, a gain coefficient of 1 corresponds to 150 lux, and a maximum gain coefficient of 1.8 corresponds to 1000 lux. Intermediate illuminance values ​​are calculated using linear interpolation. The diffuse gain coefficient is limited to the range of 0.3 to 1.8. When the ambient light intensity is below zero lux or above one thousand lux, the gain coefficient is cut off to 0.3 or 1.8 respectively to prevent the makeup material from being too dark or overexposed under extreme lighting conditions.

[0042] The mapping method for the specular reflection gain coefficient is as follows: Using an illuminance of 150 lux as the standard reference illuminance, the specular reflection gain coefficient is set to one under this illuminance condition. When the ambient illuminance is below 150 lux, the specular reflection gain coefficient decreases according to a logarithmic curve as the illuminance decreases, but at a lower rate than the diffuse reflection gain coefficient. That is, a minimum gain coefficient of 0.5 corresponds to an illuminance of 0 lux, and a gain coefficient of 1000 lux corresponds to 1. Intermediate illuminance values ​​are calculated using logarithmic interpolation. When the ambient illuminance is above 150 lux, the specular reflection gain coefficient increases according to a logarithmic curve as the illuminance increases, but the rate of increase gradually decreases. That is, a specular reflection gain coefficient of 1000 lux corresponds to 1.3, and a maximum specular reflection gain coefficient of 1.3 corresponds to 1000 lux. Intermediate illuminance values ​​are calculated using logarithmic interpolation. The specular reflection gain coefficient is limited to the range of 0.5 to 1.3, which is narrower than the range of diffuse reflection gain coefficient, in order to prevent the makeup material from producing excessive specular reflection glare in strong light conditions, while ensuring that the makeup material still retains the necessary luster in low light conditions.

[0043] In addition, in the specific implementation, the intensity of the diffuse channel of the AR makeup layer is modulated by the diffuse gain coefficient, that is, the diffuse channel value of each pixel in the AR makeup layer is multiplied by the diffuse gain coefficient; then, the intensity of the specular channel of the AR makeup layer is modulated by the specular gain coefficient, that is, the specular channel value of each pixel in the AR makeup layer is multiplied by the specular gain coefficient.

[0044] In step S4, the AR makeup effect data is displayed on the display surface of the semi-transparent and semi-reflective mirror display device.

[0045] In practical implementation, the AR makeup try-on effect data is displayed on the display surface of the semi-transparent, semi-reflective display device. Specifically, the generated AR makeup try-on effect data is written row-by-row and column-by-column according to the pixel array of the display surface. Each pixel unit in the display driver cache corresponds to a physical pixel on the display surface, and each pixel unit stores the RGB color value of that pixel. The display driver chip reads the RGB color values ​​of each pixel unit from the driver cache at a fixed refresh rate and converts them into corresponding voltage signals, driving the liquid crystal units of the transmissive display layer to emit light according to the corresponding brightness and color. The light emitted from the transmissive display layer passes sequentially through the display surface, the bonding layer, and the reflective layer, finally reaching the user's eyes. During this process, through a pre-defined spatial mapping relationship between the display surface coordinate system and the reflective surface coordinate system, the display position of the AR makeup try-on preview model is aligned pixel-by-pixel with the imaging position of the user's real facial image in the reflective surface. This ensures that the outgoing direction of the AR makeup effect light and the reflected light path of the real facial image observed by the user through the reflective surface form an optical conjugate superposition in the user's line of sight. When a user stands in front of a semi-transparent, semi-reflective display device, their eyes simultaneously receive two beams of light: one is reflected light from the user's real face, creating a mirror image; the other is transmitted light from the AR makeup effect emitted by the transmissive display layer, which penetrates the reflective layer and enters the eye, creating a virtual makeup image superimposed on the real face. These two beams of light overlap on the user's retina, allowing them to visually perceive a unified image where their real face and the virtual makeup effect are seamlessly integrated. In other words, the user can see both the contours, skin texture, and dynamic expressions of their real face, as well as the colors and shimmering effects of the AR makeup superimposed on the corresponding areas of their face.

[0046] like Figure 2The diagram shows a device scenario for a mirror-based makeup trial interactive display system integrating AR technology provided in this application. It includes a semi-transparent, semi-reflective mirror display device and a data processing unit built into the device. The semi-transparent, semi-reflective mirror display device is used to collect real-time image data of the user's face in the reflective surface of the device, posture and motion data, and current ambient light parameters. It presents a true image of the user's face on the reflective surface. The data processing unit is connected to the semi-transparent, semi-reflective mirror display device and performs 3D modeling, posture driving, AR makeup rendering, and adaptive adjustment of ambient light based on the received real-time image data, posture and motion data, and ambient light parameters. It generates AR makeup trial effect data and outputs it to the display surface of the semi-transparent, semi-reflective mirror display device, so that the AR makeup effect presented on the display surface and the user's true facial image in the reflective surface form an optical conjugate superposition in the observer's line of sight.

[0047] Therefore, this application firstly acquires real-time mirror image data of the user's face through the reflective surface of a semi-transparent, semi-reflective display device, constructs a three-dimensional facial model using multi-angle image acquisition and stereo matching technology, and simultaneously drives the model to achieve synchronized movement of overall orientation and local expressions using posture and motion data, providing users with a dynamic experience foundation for free mirroring and improving the naturalness and real-time responsiveness of the virtual makeup interaction; secondly, by attaching a two-dimensional AR makeup layer to the irregular curved surface of the three-dimensional facial model, and performing conformal stretching and compression compensation on the makeup layer according to the curvature distribution, the AR makeup presents a uniform and natural coverage effect in different areas of the face, improving the precision and realism of the virtual makeup; thirdly, by dynamically adjusting the lighting properties of the makeup material through the current ambient light parameters, the lighting response of the AR makeup is kept consistent with the lighting performance of the real facial mirror under the same ambient light, improving the naturalness of the virtual-real fusion and scene adaptability; finally, the AR makeup effect data is displayed on the display surface, allowing users to see the AR makeup effect superimposed on the corresponding position while observing their own real facial mirror, improving the intuitiveness and interactive experience of the makeup.

[0048] In summary, the technical solution adopted in this application can achieve a consistent integration between virtual makeup and real face, improving the precision and interactive experience of virtual makeup try-on in mirror try-on scenarios.

[0049] Example 2: This application provides a mirror-based makeup trial interactive display system that integrates AR technology, referencing... Figure 3 As shown in the figure, this is a block structure diagram of a mirror makeup-integrated interactive display system according to this embodiment of the present application. The interactive display system includes: The data acquisition module 100 is used to acquire real-time mirror data and posture and motion data of the user's face in the reflective surface of the semi-transparent and semi-reflective display device, construct a three-dimensional facial model based on the real-time mirror data, and drive the three-dimensional facial model to perform synchronous movement through the posture and motion data. The model building module 200 is used to respond to the user's selection instruction for the target makeup in the preset makeup library, obtain the corresponding AR makeup layer, overlay and render the AR makeup layer onto the three-dimensional facial model, and generate an AR makeup preview model. The effect generation module 300 is used to acquire the current ambient light parameters, adjust the lighting attributes of the AR makeup preview model according to the ambient light parameters, and generate AR makeup effect data. The effect display module 400 is used to display the AR makeup effect data on the display surface of the semi-transparent and semi-reflective mirror display device.

[0050] The foregoing detailed an example of a mirror-based makeup interaction display method and system integrating AR technology, as provided in the embodiments of this application. It is understood that the corresponding system includes hardware structures and / or software modules for executing each function in order to achieve the aforementioned functions. Those skilled in the art should readily recognize that, based on the units and algorithm steps described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed through hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0051] In embodiment three, this application also provides a computer device, the computer device including a memory and a processor, the memory for storing computer programs, and the processor for calling and running the computer programs from the memory, so that the computer device executes the above-described mirror makeup interactive display method integrating AR technology.

[0052] In this embodiment, reference Figure 4 The dashed lines in the figure indicate that the unit or module is optional. This figure is a structural schematic diagram of a computer device for a mirror makeup-integrated interactive display system based on an embodiment of this application. The above-described mirror makeup-integrated interactive display method based on AR technology in the above embodiment can be... Figure 4 The computer device shown is used to implement this, and the computer device includes at least one processor 501, a memory 502 and at least one communication unit 505. The computer device may be a terminal device, a server or a chip.

[0053] Processor 501 can be a general-purpose processor or a special-purpose processor. For example, processor 501 can be a central processing unit (CPU), which can be used to control computer devices, execute software programs, and process data from software programs. The computer device may also include a communication unit 505 to realize signal input (reception) and output (transmission).

[0054] For example, the computer device may be a chip, and the communication unit 505 may be the input and / or output circuit of the chip, or the communication unit 505 may be the communication interface of the chip, which may be a component of a terminal device, network device or other device.

[0055] For example, the computer device may be a terminal device or a server, and the communication unit 505 may be a transceiver of the terminal device or the server, or the communication unit 505 may be a transceiver circuit of the terminal device or the server.

[0056] The computer device may include one or more memories 502 storing a program 504. The program 504 can be executed by a processor 501 to generate instructions 503, causing the processor 501 to perform the methods described in the above method embodiments according to the instructions 503. Optionally, the memory 502 may also store data (such as a target audit model). Optionally, the processor 501 may also read data stored in the memory 502, which may be stored at the same storage address as the program 504, or the data may be stored at a different storage address than the program 504.

[0057] The processor 501 and memory 502 can be configured separately or integrated together, for example, integrated on the system-on-chip (SOC) of the terminal device.

[0058] It should be understood that each step of the above method embodiment can be completed by hardware logic circuits or software instructions in processor 501. Processor 501 can be a central processing unit, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other programmable logic device, such as discrete gate, transistor logic device, or discrete hardware component.

[0059] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] In embodiment four, this application also provides a computer-readable storage medium storing instructions or code that, when executed on a computer, cause the computer to implement the above-described mirror makeup interactive display method integrating AR technology.

[0061] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0062] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of the invention. Therefore, if these modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A mirror-based makeup application interactive display method integrating AR technology, characterized in that, The interactive display method includes the following steps: The system acquires real-time mirror data and posture data of the user's face in the reflective surface of a semi-transparent and semi-reflective display device, constructs a three-dimensional facial model based on the real-time mirror data, and drives the three-dimensional facial model to perform synchronous movement through the posture data. In response to the user's selection command for a target makeup in the preset makeup library, the corresponding AR makeup layer is obtained, and the AR makeup layer is overlaid and rendered onto the three-dimensional facial model to generate an AR makeup preview model. Obtain the current ambient light parameters, adjust the lighting attributes of the AR makeup preview model according to the ambient light parameters, and generate AR makeup effect data; The AR makeup effect data is displayed on the display surface of the semi-transparent, semi-reflective mirror display device.

2. The mirror makeup application interactive display method integrating AR technology as described in claim 1, characterized in that, The real-time mirror data includes: facial contour data, texture data, and pose data of the user's face in the reflective surface of the semi-transparent and semi-reflective mirror display device, as well as multi-angle reflection imaging data from image acquisition units in different orientations; The posture and motion data includes: the user's head position parameters and facial expression parameters. The head position parameters include head rotation angle, pitch angle, and lateral tilt angle. The facial expression parameters include the relative displacement changes between facial key points.

3. The mirror makeup application interactive display method integrating AR technology as described in claim 1, characterized in that, Constructing a 3D facial model based on the real-time mirror data, and driving the 3D facial model to perform synchronous movement using the posture and motion data, specifically includes: Two-dimensional feature point set and multi-angle reflection imaging data of the user's face are extracted from the real-time mirror data. The two-dimensional feature point set and the multi-angle reflection imaging data are stereo matched and mapped to three-dimensional space to generate an initial three-dimensional facial mesh model. Head pose parameters are extracted from the posture and motion data. A rotation transformation is applied to the initial three-dimensional facial mesh model based on the head pose parameters so that the overall orientation of the three-dimensional facial model is consistent with the user's real-time head posture. Facial expression parameters are extracted from the posture and motion data, and the vertices of the corresponding regions in the three-dimensional facial mesh model are driven to shift along the normal direction based on the facial expression parameters, so that the local deformation of the three-dimensional facial model is synchronized with the user's real-time facial expression.

4. The mirror makeup application interactive display method integrating AR technology as described in claim 1, characterized in that, The process of overlaying and rendering the AR makeup layer onto the 3D facial model to generate an AR makeup preview model specifically includes: The AR makeup layer is transformed from the image coordinate system to the local surface coordinate system of the 3D facial model through inverse projection mapping, so that the AR makeup layer fits the irregular surface shape of the 3D facial model. The compensation coefficient matrix is ​​determined based on the local surface coordinate system of the three-dimensional facial model at the corresponding location, and the compensation coefficient matrix is ​​used to apply a coordinate offset that is negatively correlated with the curvature to each pixel coordinate of the AR makeup layer; The compensated AR makeup layer is then pixel-wise fused with the corresponding parts of the 3D facial model to generate an AR makeup preview model.

5. The mirror makeup application interactive display method integrating AR technology as described in claim 1, characterized in that, The AR makeup preview model is a three-dimensional digital facial model used to represent the virtual makeup effect on the user under the target makeup look.

6. The mirror makeup application interactive display method integrating AR technology as described in claim 1, characterized in that, Adjusting the lighting attributes of the AR makeup preview model based on the ambient light parameters to generate AR makeup effect data specifically includes: The diffuse reflection intensity and specular reflection intensity of the makeup material in the AR makeup preview model are adjusted according to the light intensity and color temperature in the ambient light parameters. The AR makeup preview model's surface highlights and projection orientation are remapped based on the lighting direction in the ambient light parameters to generate AR makeup effect data. The ambient light parameters are used as real-time shading inputs in the rendering pipeline of the AR makeup preview model for pixel-by-pixel lighting calculation.

7. The mirror makeup application interactive display method integrating AR technology as described in claim 6, characterized in that, Adjusting the diffuse and specular reflection intensity of makeup materials in the AR makeup preview model based on the ambient light parameters, including the light intensity and color temperature, specifically includes: The hue offset vector of the current ambient light is determined based on the color temperature, and the original hue value of the AR makeup layer is offset by the hue offset vector to generate an environment-adaptive hue value. The diffuse reflection gain coefficient and the specular reflection gain coefficient are determined based on the light intensity. The diffuse channel intensity of the AR makeup layer is modulated using the diffuse gain coefficient; The specular reflection channel intensity of the AR makeup layer is modulated using the specular reflection gain coefficient.

8. A mirror makeup trial interactive display system integrating AR technology, used to execute the mirror makeup trial interactive display method integrating AR technology as described in any one of claims 1 to 7, characterized in that, The mirror makeup trial interactive display system integrating AR technology includes: The data acquisition module is used to acquire real-time mirror data and posture and motion data of the user's face in the reflective surface of the semi-transparent and semi-reflective display device, construct a three-dimensional facial model based on the real-time mirror data, and drive the three-dimensional facial model to perform synchronous movement through the posture and motion data; The model building module is used to respond to the user's selection command for the target makeup in the preset makeup library, obtain the corresponding AR makeup layer, overlay and render the AR makeup layer onto the three-dimensional facial model, and generate an AR makeup preview model. The effect generation module is used to acquire the current ambient light parameters, adjust the lighting attributes of the AR makeup preview model according to the ambient light parameters, and generate AR makeup effect data. The effect display module is used to display the AR makeup effect data on the display surface of the semi-transparent and semi-reflective mirror display device.

9. A computer device, characterized in that, The computer device includes a memory and a processor. The memory is used to store computer programs, and the processor is used to call and run the computer programs from the memory, so that the computer device performs a mirror makeup interactive display method integrating AR technology as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions or code that, when executed on a computer, cause the computer to implement the mirror makeup interactive display method integrating AR technology as described in any one of claims 1 to 7.

Images