Method and system for realizing personalized recommendation based on multi-feature fusion face recognition

Abstract

The invention relates to a method and a system for realizing personalized recommendation based on multi-feature fusion face recognition, and aims to provide real-time, efficient and accurate recommendation service, the method comprises the following steps: collecting a face image of a target audience in real time through a camera device, and preprocessing the image to extract a face region; running a face recognition algorithm in the edge computing device, and extracting multi-modal features including age, gender, appearance and style features; a comprehensive feature vector is generated by using a Transform model through a feature fusion module, and style classification and appearance classification are realized; according to the method, user feature vectors and a content database are matched on the basis of cosine similarity, personalized recommendation content is generated, the weight of a classification model is dynamically optimized in combination with user feedback, and dynamic behaviors and appearance features of a user are accurately captured through a time sequence analysis module, a dynamic style analysis module and a deep appearance analysis module; and comprehensively generating user feature description by combining vision, voice and character multi-mode information.

Description

Technical Field

[0001] The present invention relates to a method and system for realizing personalized recommendation based on multi-feature fusion face recognition, and in particular to a method and system for realizing personalized recommendation based on multi-feature fusion face recognition applied in the field of face recognition technology. Background Art

[0002] With the rapid development of artificial intelligence and big data technologies, face recognition technology has been widely used in personalized recommendation systems. By capturing users' facial features and behavioral data, it provides users with accurate content recommendations. However, existing technologies still have certain limitations in multimodal feature fusion, dynamic behavior analysis, and personalized recommendation optimization.

[0003] Chinese invention patent CN108550059B uses cloud devices to perform portrait detection on customer images uploaded by store photography equipment, extracts customer attribute information (such as age, gender, height) through forward calculation, and pushes customer images and attribute information to store clerk equipment to assist store clerks in making product recommendations. However, there are still certain shortcomings: Single attribute analysis: only based on static attributes such as age and gender, lacks real-time analysis of customer dynamic behaviors (such as expressions and gestures), and the recommended content is not personalized enough. Lack of multimodal fusion: does not combine multimodal information such as voice and text, making it difficult to fully characterize user characteristics, and relies on manual intervention: the recommendation process requires the participation of store clerks, making it difficult to achieve fully automated recommendations, limiting the scalability of application scenarios.

[0004] Chinese invention patent CN111415217B obtains the user's facial features through a camera, generates a user portrait based on consumption history and product information, filters and recommends products based on the matching degree between the portrait and the product, and prompts the user through a prompter. The shortcomings are: Static user portrait: The user portrait is updated based on historical consumption data and fails to combine real-time dynamic behavior or appearance change features. Insufficient feature fusion: Lacks the ability to integrate multiple features, and only makes recommendations based on a single facial feature. It cannot handle complex scenarios (such as style classification or appearance prediction). The scope of application is limited: It is mainly suitable for smart shelf environments and cannot meet the needs of multiple scenarios such as virtual social interaction and advertising.

[0005] Although a lot of exploration has been done in the existing technology, there are still certain limitations overall: for example, the ability to capture dynamic features is insufficient: the existing technology mainly makes recommendations based on static features (such as age, gender, shopping history), lacks real-time analysis of user dynamic behaviors (such as expression changes, posture adjustments), and lacks multimodal feature fusion capabilities: it only relies on a single feature (such as image data) for recommendations, does not combine multimodal information such as voice and text, and cannot fully characterize user characteristics, and lacks cross-domain application capabilities: existing systems are mostly limited to the retail field, and lack support for extended applications such as virtual image generation or health management. Summary of the invention

[0006] In view of the above-mentioned prior art, the technical problem to be solved by the present invention is how to combine the style classification method of time series analysis to capture the user's dynamic behavior and subtle changes in expression in real time, generate a comprehensive feature vector by fusing visual, voice and text features, comprehensively characterize user characteristics, and optimize the recommendation accuracy by dynamically adjusting the classification model weights and user feedback.

[0007] To solve the above problems, the present invention provides a method and system for realizing personalized recommendation based on multi-feature fusion face recognition, comprising the following steps:

[0008] S1: The facial image of the target audience is collected in real time through the camera device, and the image data is transmitted to the image processing module of the edge computing device to pre-process the image to extract the face area;

[0009] S2: Run the face recognition algorithm in the edge computing device to extract multiple characteristics of the target audience, including age characteristics, gender characteristics, appearance characteristics, and style characteristics, among which:

[0010] S2.1: Extract age features through age analysis unit;

[0011] S2.2: Extract gender characteristics through gender analysis unit;

[0012] S2.3: The appearance analysis unit extracts geometric features (nose bridge height, chin curve), texture features (skin glossiness, pore distribution) and generates aesthetic scores based on symmetry, proportion coordination, light perception and other dimensions;

[0013] S2.4: The style analysis unit combines time series analysis of user behavior (such as smiles, gestures, sitting postures) and expression changes, combines the generative adversarial network (GAN) to generate diversified style data in the target scene, and combines multimodal information (vision, voice, text) to generate style feature vectors;

[0014] S3: The feature fusion module uses the multi-head attention mechanism of the Transformer model to capture the interaction between global and local features, generate a 512-dimensional comprehensive feature vector, and share the underlying feature extraction layer to achieve style and appearance classification;

[0015] S4: The recommendation matching module calculates the matching degree between the user feature vector and the feature vector of the content database based on cosine similarity, generates personalized recommendation content, and dynamically adjusts the recommendation results based on makeup features;

[0016] S5: Dynamically optimize the feature classification model weights by combining the recommendation optimization module with user feedback data, and improve the generalization ability of the classification model through self-supervised learning, thereby continuously optimizing the recommendation accuracy;

[0017] S6: Push the finally generated recommended content to the electronic display screen to display personalized content for different audiences.

[0018] In the above-mentioned method and system for realizing personalized recommendation based on multi-feature fusion face recognition, the system collects user facial images in real time through camera equipment, extracts multiple features including age, gender, appearance and style features, combines time series analysis with multimodal fusion technology, and generates comprehensive feature vectors through the Transformer model for personalized content recommendation. The recommendation model is optimized through a dynamic optimization module combined with user feedback and self-supervised learning to improve the accuracy and adaptability of the recommended content.

[0019] As a further improvement of this application, the style analysis module includes:

[0020] Dynamic behavior unit: used to analyze user behavior (such as smile, gesture, sitting posture) based on time series;

[0021] Expression change unit: The multi-head attention mechanism based on the Transformer model captures the relationship between subtle changes in expression and movement trajectory;

[0022] Multimodal fusion unit: First, extract the user's overall temperament characteristics through visual analysis, and then combine the voice intonation and text input information to generate a comprehensive style feature vector.

[0023] As a further improvement of this application, the appearance analysis module includes:

[0024] 3D modeling unit: Reconstructs the user's facial geometry and proportions, including nose bridge height, chin curve, etc., based on convolutional neural network (CNN);

[0025] Deep texture unit: extracts microscopic features such as skin glossiness and pore distribution through local binary patterns (LBP);

[0026] Aesthetic scoring unit: Generates a comprehensive aesthetic score based on weights of dimensions such as symmetry, proportion coordination, and light perception;

[0027] Social perception unit: Combines users’ social data (such as likes and comments) to optimize appearance rating results to reflect group recognition.

[0028] As a further improvement of the present application, the feature fusion module includes:

[0029] Feature vector generation unit: uses the multi-head attention mechanism of the Transformer model to capture global and local feature interactions;

[0030] Style classification unit: captures subtle changes in time series and generates dynamic style features;

[0031] Appearance classification unit: integrates facial features proportions, skin features and overall contours through a global attention mechanism;

[0032] 7. Multi-task learning unit: Shares the underlying feature extraction layer and completes style classification and appearance classification tasks at the same time, improving classification efficiency and reducing computing resource consumption.

[0033] 8. As another improvement of the present application, the recommendation matching module calculates the cosine similarity between the user feature vector and the content feature vector by the following formula:

[0034]

[0035] Among them, A is the user feature vector, and B is the content database feature vector.

[0036] As another improved supplement of the present application, it is recommended that the optimization module includes:

[0037] Appearance prediction unit: predicts the user's skin condition and appearance change trend based on the time series analysis model;

[0038] Makeup suggestion unit: Generates personalized retouching suggestions based on appearance prediction results, including adjusting foundation suggestions based on skin radiance and pore distribution;

[0039] Model optimization unit: Dynamically adjusts the feature classification model weights based on user feedback and optimizes model performance through self-supervised learning.

[0040] As another improved supplement of the present application, the image processing module: is used for real-time acquisition and pre-processing of facial images of the target audience;

[0041] Feature analysis module: including age analysis unit, gender analysis unit, appearance analysis unit and style analysis unit;

[0042] Feature fusion module: Generates multi-feature fusion vectors based on the Transformer model;

[0043] Recommendation matching module: used to calculate the similarity between user features and content features;

[0044] Recommendation optimization module: used to dynamically adjust recommendation content and optimize classification models.

[0045] As another improvement of the present application, the feature analysis module further includes:

[0046] Dynamic style analysis module: used to extract user dynamic style by combining time series analysis and multimodal information;

[0047] Deep appearance analysis module: used for 3D modeling and deep texture analysis to generate appearance scores;

[0048] Social perception module: optimize appearance rating results through user social data.

[0049] The system supports cross-domain applications, including:

[0050] A virtual image generation unit: used to generate a virtual avatar in a virtual social or metaverse platform based on the style classification results;

[0051] Medical assistance unit: used to detect skin problems or health indicators (such as facial edema).

[0052] Edge computing devices include:

[0053] Hardware support module: supports GPU acceleration and large-capacity storage for efficient operation of Transformer models;

[0054] Self-supervised learning module: Automatically mine potential information in unlabeled data to optimize classification models.

[0055] In summary, this application has the following beneficial effects:

[0056] 1. Real-time and efficient data processing. By utilizing the collaborative work of camera devices and edge computing devices, the system can collect user facial images in real time and efficiently extract facial areas through fast image preprocessing algorithms (such as bilateral filtering and normalization). Edge computing devices support GPU parallel computing and are combined with large-capacity storage modules to improve the speed of multimodal feature processing and meet high-performance requirements in real-time scenarios.

[0057] 2. Comprehensiveness and accuracy of multi-feature analysis. Through time series analysis and the multi-head attention mechanism of the Transformer model, the system can capture the dynamic features of user behavior (such as smiles, gestures) and subtle changes in expressions, thereby improving the accuracy of style analysis. The system combines three-dimensional modeling with the local binary pattern (LBP) algorithm to comprehensively extract the user's geometric features (such as nose bridge height, chin curve) and texture features (such as skin gloss and pore distribution), and generates an intuitive and reliable appearance score through the aesthetic scoring unit. The system integrates visual, voice and text features to generate a comprehensive style feature vector, covering the user's dynamic behavior, language expression and emotional state, making the analysis results more comprehensive and rich.

[0058] 3. Personalization and adaptability of recommended content. The system uses the multi-head attention mechanism of the Transformer model to fuse multimodal features into a 512-dimensional comprehensive feature vector, capture the interactive relationship between global and local features, and ensure the accuracy of recommended content. Based on user feedback (such as click-through rate and rating), the system dynamically adjusts the weight parameters of the recommendation model, and further optimizes the model performance through self-supervised learning to improve the adaptability of personalized recommendations. The system can dynamically adjust the display order and style of recommended content, which is suitable for various scenarios such as advertising and user experience optimization.

[0059] 4. Cross-domain expansion capabilities. The system generates virtual images based on style classification results, which is suitable for personalized interactive scenarios in virtual social or metaverse platforms. The virtual images support real-time dynamic interaction (such as expression synchronization and action response). The system detects the user's skin health status (such as pore distribution, wrinkle depth) and potential health problems (such as facial edema) through appearance feature analysis, and provides users with skin care advice and health management support.

[0060] 5. Self-optimization capability of model performance. The system dynamically optimizes the recommendation model based on user behavior data. By adjusting the weights and recommendation strategies of the classification model, it continuously improves the accuracy of recommendation results and user satisfaction. The system utilizes the potential information in unlabeled data and generates a pre-trained model through comparative learning, which improves the generalization ability of the model and significantly reduces the dependence on large amounts of labeled data.

[0061] 6. Flexibility and robustness of the system architecture. The system consists of image processing modules, feature analysis modules, feature fusion modules, recommendation optimization modules, etc. Each module is independent in function and works together efficiently, making it easy to expand and maintain. The system maintains a high level of feature extraction accuracy in complex scenarios such as light changes and occlusions through normalization, deep learning, and multimodal fusion. The system is compatible with a variety of edge computing devices, and it is recommended to use GPUs for high-performance computing to meet the needs of different computing resource environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 System interaction for this application Figure 1 ;

[0063] Figure 2 System interaction for this application Figure 2 ;

[0064] Figure 3 System interaction for this application Figure 3 . DETAILED DESCRIPTION

[0065] Three embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0066] Embodiment 1:

[0067] like Figures 1 to 3 As shown, this embodiment relates to a method and system for realizing personalized recommendation based on multi-feature fusion face recognition, describes the whole process from data collection to personalized recommendation content generation and display, and focuses on the basic functions and core architecture of the system.

[0068] System construction and hardware configuration

[0069] Camera device installation: Install the camera device at a location with a wide field of view near the electronic display screen to ensure that the facial image of the target audience can be clearly captured. The camera device is connected to the edge computing device via a USB interface or an HDMI interface and used as an image acquisition source.

[0070] Edge computing device configuration: Initialize the network settings of the edge computing device so that it can operate normally in the local network environment, install face recognition related algorithm software, including age feature detection algorithm, gender feature detection algorithm, appearance feature detection algorithm and style feature detection algorithm, and configure algorithm parameters, such as adjusting the sensitivity and accuracy of the algorithm according to the actual application scenario.

[0071] Electronic display screen configuration: Connect the electronic display screen to the edge computing device to establish a data transmission channel for receiving recommended content and displaying it.

[0072] Working process

[0073] S1: Data collection and preprocessing

[0074] The camera device collects facial images of the target audience in real time and transmits them to the edge computing device at a frame rate of 30 frames per second. The image processing module pre-processes the collected images, including denoising, cropping and normalization, to extract the facial area.

[0075] S2: Feature extraction

[0076] Age feature extraction (S2.1): Use a deep convolutional neural network (CNN) to extract facial features, input the extracted features into the support vector regression (SVR) model, and predict the age value using the following formula:

[0077] y=WX+b

[0078] Among them, y is the predicted age value, W is the weight matrix, X is the feature vector, and b is the bias term.

[0079] Gender feature extraction (S2.2): Extract facial geometric features (such as eyebrow shape and eye size) and texture features (such as skin texture), and use the support vector machine (SVM) classifier to predict gender. The decision function is:

[0080] f(x)=sign(W T X+b)

[0081] Wherein, sign is the sign function. When wTx+b>0, f(x)=1, indicating that the prediction is male; when wTx+b<0, f(x)=-1, indicating that the prediction is female.

[0082] Appearance feature extraction (S2.3): Geometric features (such as nose bridge height and chin curve) are extracted using a key point detection algorithm, and texture features (such as skin gloss and pore distribution) are extracted using a local binary pattern (LBP) algorithm. An aesthetic score is generated based on dimensions such as symmetry, proportion coordination, and light perception, ranging from 1 to 10 points.

[0083] First, image preprocessing is performed. The face image is adjusted to a uniform size and grayscaled. Simple filtering is used to remove noise. Then geometric feature extraction is performed. A fast key point detection algorithm is used to determine the positions of key feature points of the face, such as the eyes, nose, and mouth. The distance characteristics between the main facial organs, such as the distance between the eyes and the distance between the nose and mouth, are calculated based on the coordinates of these key points. The key points are also used to roughly determine the contours of the main organs and calculate their area characteristics. Simple normalization is performed. Finally, texture feature extraction is performed. The central pixel is used as the threshold in the local area of ​​the image, and a simple LBP value is generated by comparing it with adjacent pixels. The LBP features in a certain area are counted. Through these steps, geometric and texture features that can reflect the appearance characteristics of the face are extracted, providing a basis for subsequent appearance scoring.

[0084] Face Prediction:

[0085] For a new face image, the feature vector xnew is obtained through the feature extraction step, and it is input into the trained SVM model, that is, f(xnew)=wT*(xnew)+b is calculated to obtain the predicted value of the appearance score.

[0086] Style feature extraction (S2.4): Combine time series analysis with user behavior (such as smiles, gestures, sitting postures) and expression changes, use generative adversarial networks (GANs) to generate diverse style data, and fuse multimodal information (vision, speech, text) to generate style feature vectors.

[0087] Recognition algorithm of facial style features

[0088] Extract facial geometric features:

[0089] The position and distance of key facial points: including the position coordinates of key parts such as eyes, eyebrows, nose, mouth, chin, as well as the distances between these parts, the distance between the two eyes, and the vertical distance from the eyes to the mouth.

[0090] Facial contour shape: describes the overall contour shape of the face, face shape (round, square, oval), the curve of the face edge, etc.

[0091] Color Features:

[0092] Skin color: In a specific color space (such as HSV, YCbCr, etc.), human skin color has a certain distribution range. By converting the image from the RGB color space to the target color space, the face area can be extracted or used as one of the features based on the distribution characteristics of the skin color in the space.

[0093] Hair color: Hair color is also an important feature. Different people may have different hair colors, such as black, brown, blonde, etc.

[0094] The normalized training set face images are partitioned by style features, the LBP features of each face style area are calculated, and the feature vectors of the area are integrated using the histogram statistics method to form a training set local feature library composed of local features of specific faces; for the test faces, the face images are normalized, the faces are partitioned, the local LBP features are calculated, and the local histogram statistics operations are performed in the same way; finally, for the local histogram statistical features of the test faces, the training set feature library is used for local sparse reconstruction representation, and the local sparse reconstruction residual weighted method is used for the final face style classification and recognition.

[0095] Local Binary Pattern (LBP) is an operator used to describe the local texture features of an image. The calculation formula of LBP features is as follows:

[0096]

[0097] Where R is the neighborhood radius, which is usually 1 (corresponding to a 3*3 neighborhood), and s(x) is the sign function, defined as:

[0098]

[0099] S3: Feature fusion, using the multi-head attention mechanism of the Transformer model to capture the global and local interactions between features, generate a unified 512-dimensional feature vector, and share the underlying feature extraction layer to achieve style and appearance classification.

[0100] In the process of personalized recommendation, the various feature information collected is effectively transformed and encoded and fused in the form of feature vectors. For the age, gender, appearance, and style of the face, multiple features are fused together through a vector embedding model. This model can convert each feature into a 512-dimensional vector representation, and then combine these vectors through a fusion algorithm to form a fusion vector that comprehensively reflects multiple features. Such a fusion vector not only retains the unique information of each feature, but also captures the potential correlation between different features, providing a richer and more accurate basis for subsequent analysis and recommendations.

[0101] Convert the collected feature information into a format that can be analyzed and encode and fuse it in the form of feature vectors:

[0102] Milti-Feature Fusion Vector=Embedding512[age,gender,style,appearancescore]

[0103] S4: Recommended content generation, the recommendation matching module uses the following formula to calculate the cosine similarity between the user feature vector and the content feature vector:

[0104]

[0105] Where A and B are two eigenvectors, representing the dot product of the vectors, and ‖A‖ and ‖B‖ represent the modulus of the vectors.

[0106] S6: Recommended content delivery: recommended content is pushed to the electronic display screen by the edge computing device, and the content display order and style are dynamically adjusted according to user characteristics to achieve personalized content delivery.

[0107] Face region extraction method: Haar feature cascade classifier or deep learning-based MTCNN framework is used to detect faces in images. In the detected face region, bilateral filtering algorithm is applied for denoising, and normalization technology is used to adjust the pixel value range to [0,1] to improve the robustness of the model to different lighting conditions.

[0108] Dynamically adjust scene description: Support content synchronization on multiple screen devices. For example, in a billboard system, multiple screens display personalized advertisements that match different users, adjust the display time and order of advertisement content according to the real-time number of viewers, and optimize the delivery strategy through audience data.

[0109] This embodiment uses a camera to collect facial images of the target audience in real time, and uses edge computing devices to quickly process data to achieve real-time delivery of recommended content.

[0110] With the help of CNN and Transformer models, the system can accurately extract multiple characteristics of users (age, gender, appearance, style) and fuse them to generate a comprehensive feature vector, thereby improving the matching degree of recommended content.

[0111] The system supports applications in a variety of scenarios (such as advertising and user experience optimization), and adapts to different hardware environments through parameter adjustment.

[0112] Example 1 demonstrates the basic architecture and functions of the system. It realizes a closed-loop process from input to recommendation through the integration of hardware configuration, data collection and feature extraction, laying the foundation for the subsequent expansion of personalized recommendation functions.

[0113] Embodiment 2:

[0114] like Figures 1 to 3 As shown, based on Example 1, this example describes the implementation method based on the dynamic style analysis module and the deep appearance analysis module, focusing on the technical details of the system in terms of user dynamic behavior, expression changes, appearance feature extraction and multimodal information fusion, and showing how to combine multiple features through the feature fusion module to generate a comprehensive feature vector.

[0115] Style Analysis Module

[0116] Implementation method of dynamic behavior unit: Use camera equipment to collect the target audience's behavioral action data in real time, including dynamic information such as smiles, gestures, and sitting postures. Use time series analysis technology to extract behavioral features, such as smile duration, gesture change frequency, and sitting posture stability. Time series data is processed in frames, and the action state changes of each frame are analyzed. The action changes are encoded through feature vectors to form dynamic behavior features.

[0117] Implementation method of expression change unit: Use the multi-head attention mechanism of the Transformer model to capture the correlation between subtle changes in expression and movement trajectory, extract the time series data of facial key points (such as eyebrow raising angle, mouth opening and closing amplitude), and embed the expression features into vector representation to capture the interaction of facial key points over time, and output the expression dynamic feature vector as an important part of style features.

[0118] Implementation method of multimodal fusion unit: extract the user's overall temperament characteristics (such as maturity and sweetness), and perform deep learning on facial texture information through convolutional neural network (CNN), collect user voice intonation, speaking speed and other features, convert them into feature vectors through spectral analysis, combine with user input text, and use natural language processing (NLP) model to extract language style. Visual, voice and text features are encoded as independent vectors respectively, and multimodal information is fused using Transformer model to generate a comprehensive style feature vector.

[0119] Appearance analysis module

[0120] Implementation method of the 3D modeling unit: Reconstruct the 3D model of the user's face based on the convolutional neural network (CNN), extract the facial geometry and proportion features, including the height of the nose bridge, the chin curve and the face contour, use the depth map to build a 3D point cloud model, and extract the spatial relationship features of key parts such as the nose bridge, mouth and eyes.

[0121] Implementation method of deep texture unit: Use the local binary pattern (LBP) algorithm to extract texture features such as skin gloss and pore distribution, count the local texture feature histogram and normalize it, and generate a binary pattern by comparing the pixel values ​​of the local area with the central pixel. The texture features are represented by a regional histogram, which is used to quantify the skin fineness and uniformity of pore distribution.

[0122] Implementation method of the aesthetic scoring unit: Generate aesthetic scores based on dimensions such as facial symmetry, proportion coordination, and light perception. Use a preset weight model to weight the features of each dimension. The score range is 1 to 10. Symmetry is calculated by the distance difference between the key points on both sides, and proportion coordination is verified based on the golden ratio model.

[0123] Implementation method of social perception unit: Use users' social data (such as likes and comments) to optimize the appearance score results, modify the weight distribution of the score according to group feedback, count the frequency of users' social interactions, build a weight model, and combine the aesthetic score to generate an optimized appearance score.

[0124] Feature fusion module

[0125] Implementation of the feature vector generation unit: The multi-head attention mechanism of the Transformer model is used to capture the interactive relationship between global and local features, and the age, gender, appearance and style features are embedded in a 512-dimensional comprehensive feature vector. The multi-head attention mechanism captures the potential correlation between features and outputs a unified feature vector for subsequent analysis.

[0126] Implementation method of style classification unit: dynamically capture the style change characteristics in the time series, generate style feature vectors, analyze the behavior and expression characteristics at different time points, and output dynamic style classification results.

[0127] Implementation method of appearance classification unit: Comprehensively analyze the proportion of facial features, skin texture and overall contour, and integrate local features with overall features through Transformer global attention mechanism.

[0128] Implementation method of multi-task learning unit: share the underlying feature extraction layer to realize parallel calculation of style classification and appearance classification, use the shared layer to reduce computing resources and improve classification efficiency.

[0129] Working process

[0130] The data is collected by the camera and transmitted to the edge computing device.

[0131] The style analysis module and the appearance analysis module extract the user's dynamic style features and static appearance features respectively.

[0132] The feature fusion module embeds multimodal features into a unified vector representation and performs classification and fusion.

[0133] Output comprehensive feature vectors to provide support for subsequent generation of recommended content.

[0134] The dynamic behavior unit uses LSTM (Long Short-Term Memory Network) to model user behavior sequences, capture the temporal dynamic patterns of smiles or gestures, and optimizes the learning rate and number of hidden layer nodes of LSTM to balance accuracy and computational overhead. Time series analysis using LSTM is superior to the traditional sliding window method and can more accurately predict complex behavioral changes.

[0135] The multimodal fusion unit detects emotions in the user's voice intonation, such as pleasure, happiness or anxiety. After the user enters a text comment, semantic features are extracted through the natural language processing (NLP) model, such as keyword extraction (such as "like", "recommend").

[0136] The social perception unit obtains users' likes, comments, and sharing behaviors through social media APIs, weights the interaction data, extracts the main patterns of high-frequency interactive behaviors through principal component analysis (PCA), and adjusts the scoring weights according to their contribution.

[0137] The multi-task learning unit adds a task-specific attention mechanism after the shared feature layer and avoids interference between tasks through independent loss functions.

[0138] Through dynamic behavior units and expression change units, users' dynamic behaviors and expression details can be captured in real time to improve the accuracy of style analysis.

[0139] Comprehensive analysis and integration of visual, voice and text features to generate more accurate style features.

[0140] The combination of 3D modeling and deep texture analysis provides refined appearance feature extraction capabilities and optimizes appearance scores.

[0141] The Transformer model is used to realize the interaction between global and local features, improve the efficiency of feature fusion, and reduce the consumption of computing resources.

[0142] Example 2 demonstrates the details of feature analysis and fusion through the collaborative work of dynamic style analysis and deep appearance analysis modules, highlights the technical advantages of the system in multimodal feature extraction and deep learning, and provides an accurate and efficient feature foundation for personalized recommendations.

[0143] Embodiment 3:

[0144] like Figures 1 to 3 As shown, this embodiment describes the cross-domain applications of recommendation optimization module, virtual image generation and medical assistance based on the feature extraction and analysis results of Embodiment 1 and Embodiment 2, and also demonstrates the implementation details of the system in hardware support and self-supervised learning optimization model.

[0145] Recommended optimization module

[0146] Implementation of the appearance prediction unit: Based on the time series analysis model, the skin condition changes (such as glossiness and pore distribution) of the user in multiple image collections are analyzed to predict skin aging trends or short-term condition changes (such as water loss and dry skin). The time series analysis is combined with the convolutional neural network (CNN) to extract skin texture features, capture skin changes in a specific time period, and predict skin change trends through a linear regression model:

[0147] Yt=aXT+b

[0148] Among them, Yt is the predicted value, Xt is the feature input, a and b are the regression coefficient and bias term respectively.

[0149] Implementation method of makeup suggestion unit: Provide personalized modification suggestions based on appearance prediction results, including recommending suitable foundation types (such as matte type, moisturizing type) and skin care suggestions (such as hydration, oil control), and adjust makeup suggestions according to the following rules based on skin gloss and pore distribution characteristics: high gloss and uneven pore distribution: matte foundation is recommended; low gloss and even pore distribution: moisturizing foundation is recommended. Use support vector machine (SVM) to classify makeup suggestion types.

[0150] Model optimization unit implementation method: Use user feedback (such as content click-through rate, comments, and ratings) to dynamically adjust the classification model weights, introduce self-supervised learning to optimize model performance, and mine potential information in unlabeled data. User feedback updates the model weights through weighted averaging:

[0151] Wnew=Wold+α·Feedback

[0152] Among them, α is the feedback weight adjustment factor.

[0153] Self-supervised learning generates pre-trained models through contrastive learning, improving the utilization of unlabeled data.

[0154] Cross-domain applications

[0155] Implementation method of the virtual image generation unit: Generate a virtual avatar based on the user style classification results for use in social platforms or metaverse applications. The characteristics of the virtual avatar include facial expressions, dynamic movements and overall style. Use a generative adversarial network (GAN) to generate diverse expressions of the virtual image, and adjust the dynamic behavior of the virtual avatar (such as smiling, blinking) in combination with the user style feature vector.

[0156] Implementation of the medical assistance unit: Analyze the user's skin problems (such as dryness and large pores) through deep texture analysis, use the appearance prediction function to detect health indicators (such as facial edema), input skin condition characteristics into the health model, and evaluate the skin health level. Facial edema detection is completed by calculating the displacement of key points. The formula is:

[0157] ΔD=|D current -D beaseline |

[0158] Among them, D current and D baseline are the distances between the current and reference key points, respectively.

[0159] The makeup suggestion unit supplements the scene adaptation method: in cosmetics recommendation, skin care suggestions are generated according to the user's skin type (such as dry or oily). In advertising display, dynamic display samples (such as recommended videos or 3D models) are generated based on the makeup effects with high user ratings. The appropriate cosmetics type is dynamically selected through the SVM classifier based on skin texture features. Experimental results show that the classification accuracy rate reaches 92%.

[0160] The model optimization unit constructs positive and negative sample comparison pairs: positive samples are images collected by the same user at different times, and negative samples are images of different users. The InfoNCE loss function is used to optimize the contrastive learning embedding model. The formula is as follows:

[0161]

[0162] Among them, z i and z j is the embedding vector of the positive sample pair, and τ is the temperature parameter.

[0163] Virtual image generation supplements dynamic interactive functions: real-time capture of user expressions and synchronization to the virtual image, such as blinking and smiling, supports users to personalize the appearance and behavior patterns of the virtual image, and realizes virtual social interaction through the Metaverse platform.

[0164] GAN generation effect: The StyleGAN model is used to generate the expression changes of the virtual image. The generation effect is realistic and can highly match the user's actual expression.

[0165] Medical assistance supplements health indicator detection: Facial edema detection evaluates fluid retention through the movement of key points (such as eyes and cheekbone areas), supplements skin aging indicator analysis (such as wrinkle depth and pigmentation), and is evaluated in combination with the time series model of the appearance prediction unit.

[0166] Application scenarios:

[0167] Health management: Generate user skin condition trend reports and provide skin care recommendations.

[0168] Medical assistance: Detect early health problems through skin characteristics (such as diabetic skin symptoms).

[0169] Hardware support module GPU acceleration allocation strategy: the matrix multiplication task of the attention mechanism is allocated to the GPU parallel computing, and the embedding vector generation and storage tasks are completed by the CPU and storage module in collaboration to reduce latency.

[0170] System hardware support

[0171] Hardware support module implementation method: Use GPU acceleration module to support large-scale parallel computing of Transformer model, equipped with large-capacity storage for storing multimodal feature data and model parameters. The core tasks of GPU include accelerated computing of multi-head attention mechanism. Data storage is managed in a hierarchical manner, and feature data and historical records are stored independently.

[0172] Implementation method of self-supervised learning module: automatically mine potential information in unlabeled data, generate pre-trained models, apply contrastive learning algorithm to unsupervised data, optimize embedding vectors by comparing positive and negative samples, self-supervised learning process: generate pseudo labels for unlabeled data, construct positive and negative sample comparison pairs, calculate similarity, and update embedding model parameters.

[0173] Working process

[0174] Recommendation optimization: After the recommendation content is generated, the classification model weights are adjusted based on user feedback, and self-supervised learning is used to optimize the feature utilization of unlabeled data.

[0175] Cross-domain applications: Generate a virtual image based on the user's style characteristics and upload it to a social or metaverse platform to analyze the user's skin condition and output health reports and makeup suggestions.

[0176] Hardware collaboration: GPU supports the operation of Transformer models, and edge computing devices complete recommendation optimization and feature updates in real time.

[0177] The appearance prediction and makeup suggestion modules are combined to provide personalized recommendations based on changes in user appearance and enhance the user experience.

[0178] It supports virtual image generation and medical assistance applications, expanding the application scenarios of the system.

[0179] The model is dynamically optimized based on user feedback, and the utilization rate of unlabeled data is improved through self-supervised learning.

[0180] Hardware support ensures efficient operation, especially for multi-task learning and complex feature fusion processes.

[0181] Example 3 demonstrates the system's recommendation optimization module and cross-domain application functions, highlighting the technical advantages of appearance prediction, virtual image generation and medical assistance, while combining hardware support and self-supervised learning optimization to improve the system's performance and applicability.

[0182] In view of current practical needs, the above-mentioned implementation mode adopted in this application is not limited to the scope of protection. Various changes made within the knowledge scope of technical personnel in this field without departing from the concept of this application still fall within the scope of protection of the present invention.

Claims

1. A method for realizing personalized recommendation based on multi-feature fusion face recognition, characterized in that: The following steps are involved: S1: The facial image of the target audience is collected in real time through the camera device, and the image data is transmitted to the image processing module of the edge computing device to pre-process the image to extract the face area; S2: Run the face recognition algorithm in the edge computing device to extract multiple characteristics of the target audience, including age characteristics, gender characteristics, appearance characteristics, and style characteristics, where: S2.1: Extract age features through age analysis unit; S2.2: Extract gender characteristics through gender analysis unit; S2.3: The appearance analysis unit extracts geometric features (nose bridge height, chin curve), texture features (skin glossiness, pore distribution) and generates aesthetic scores based on symmetry, proportion coordination, light perception and other dimensions; S2.4: The style analysis unit combines time series analysis of user behavior (such as smiles, gestures, sitting postures) and expression changes, combines the generative adversarial network (GAN) to generate diversified style data in the target scene, and combines multimodal information (vision, voice, text) to generate style feature vectors; S3: The feature fusion module uses the multi-head attention mechanism of the Transformer model to capture the interaction between global and local features, generate a 512-dimensional comprehensive feature vector, and share the underlying feature extraction layer to achieve style and appearance classification; S4: The recommendation matching module calculates the matching degree between the user feature vector and the feature vector of the content database based on cosine similarity, generates personalized recommendation content, and dynamically adjusts the recommendation results based on makeup features; S5: Dynamically optimize the feature classification model weights by combining the recommendation optimization module with user feedback data, and improve the generalization ability of the classification model through self-supervised learning, thereby continuously optimizing the recommendation accuracy; S6: Push the finally generated recommended content to the electronic display screen to display personalized content for different audiences.

2. The method for realizing personalized recommendation based on multi-feature fusion face recognition according to claim 1, characterized in that: The style analysis module includes: Dynamic behavior unit: used to analyze user behavior (such as smile, gesture, sitting posture) based on time series; Expression change unit: The multi-head attention mechanism based on the Transformer model captures the relationship between subtle changes in expression and movement trajectory; Multimodal fusion unit: First, extract the user's overall temperament characteristics through visual analysis, and then combine the voice intonation and text input information to generate a comprehensive style feature vector.

3. The method for realizing personalized recommendation based on multi-feature fusion face recognition according to claim 1, characterized in that: The appearance analysis module comprises: 3D modeling unit: Reconstructs the user's facial geometry and proportions, including nose bridge height, chin curve, etc., based on convolutional neural network (CNN); Deep texture unit: extracts microscopic features such as skin glossiness and pore distribution through local binary patterns (LBP); Aesthetic scoring unit: Generates a comprehensive aesthetic score based on weights of dimensions such as symmetry, proportion coordination, and light perception; Social perception unit: Combines users’ social data (such as likes and comments) to optimize appearance rating results to reflect group recognition.

4. The method for realizing personalized recommendation based on multi-feature fusion face recognition according to claim 1, characterized in that: The feature fusion module includes: Feature vector generation unit: uses the multi-head attention mechanism of the Transformer model to capture global and local feature interactions; Style classification unit: captures subtle changes in time series and generates dynamic style features; Appearance classification unit: integrates facial features proportions, skin features and overall contours through a global attention mechanism; Multi-task learning unit: Shares the underlying feature extraction layer and completes style classification and appearance classification tasks at the same time, improving classification efficiency and reducing computing resource consumption.

5. The method for realizing personalized recommendation based on multi-feature fusion face recognition according to claim 1, characterized in that: The recommendation matching module calculates the cosine similarity between the user feature vector and the content feature vector using the following formula: Among them, A is the user feature vector, and B is the content database feature vector.

6. The method for realizing personalized recommendation based on multi-feature fusion face recognition according to claim 1, characterized in that: The recommendation optimization module includes: Appearance prediction unit: predicts the user's skin condition and appearance change trend based on the time series analysis model; Makeup suggestion unit: Generates personalized retouching suggestions based on appearance prediction results, including adjusting foundation suggestions based on skin radiance and pore distribution; Model optimization unit: Dynamically adjusts the feature classification model weights based on user feedback and optimizes model performance through self-supervised learning.

7. A system for personalized recommendation based on multi-feature fusion face recognition, characterized in that: include: Image processing module: used to collect and pre-process facial images of target audiences in real time; Feature analysis module: including age analysis unit, gender analysis unit, appearance analysis unit and style analysis unit; Feature fusion module: Generates multi-feature fusion vectors based on the Transformer model; Recommendation matching module: used to calculate the similarity between user features and content features; Recommendation optimization module: used to dynamically adjust recommendation content and optimize classification models.

8. The system for implementing personalized recommendation based on multi-feature fusion face recognition according to claim 7, characterized in that: The feature analysis module further comprises: Dynamic style analysis module: used to extract user dynamic style by combining time series analysis and multimodal information; Deep appearance analysis module: used for 3D modeling and deep texture analysis to generate appearance scores; Social perception module: optimize appearance rating results through user social data.

9. The system for realizing personalized recommendation based on multi-feature fusion face recognition according to claim 7, characterized in that: The system supports cross-domain applications, including: A virtual image generation unit: used to generate a virtual avatar in a virtual social or metaverse platform based on the style classification results; Medical assistance unit: used to detect skin problems or health indicators (such as facial edema).

10. The system for realizing personalized recommendation based on multi-feature fusion face recognition according to claim 7, characterized in that: The edge computing device comprises: Hardware support module: supports GPU acceleration and large-capacity storage for efficient operation of Transformer models; Self-supervised learning module: Automatically mine potential information in unlabeled data to optimize classification models.

Images