User portrait automatic expansion method and system based on multi-modal large model
Patent Information
- Application Number
- CN202611011486.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-08
- Publication Date
- 2026-10-20
AI Technical Summary
[0002]标签维度有限，难以覆盖真实用户复杂性 传统标签系统依赖人工预定义规则（如“收听歌曲>20首=爱听歌曲”），仅能捕捉显性、结构化数据，无法处理非结构化数据（如语音、图像）中的深层信息
[0036]与现有技术相比，本发明的有益效果是：多模态大模型驱动的用户画像自动扩标系统，通过融合非结构化数据、动态生成语义标签、实现意图推理与预测，弥补了传统标签系统维度单一、更新滞后、预测能力弱的三大短板；多模态大模型可统一解析文本、图像、语音、视频、传感器等非结构化信息，极大拓展标签维度；大模型支持实时行为序列建模，实现标签的秒级动态刷新，基于Transformer的时序模型可捕捉用户行为流，自动识别动态标签；大模型能通过行为叙事建模与知识图谱融合，预测用户未来可能的需求，使画像系统从被动响应迈向主动预判；从建模到可解释的用户画像扩标，大模型不仅能生成标签，还能用自然语言解释标签成因，降低业务使用门槛；通过深度理解用户的多模态行为，自动生成更丰富、更精准、更具前瞻性的用户画像。
Smart Images
Figure FT_1 
Figure FT_2
Abstract
Description
Technical Field
[0001] This invention relates to the field of data information processing technology, and in particular to an automatic expansion method and system for user profiles based on a multimodal large model. Background Technology
[0002] Limited tag dimensions make it difficult to capture the complexity of real users. Traditional tagging systems rely on manually predefined rules (such as "listening to more than 20 songs = liking to listen to songs"), which can only capture explicit, structured data and cannot process the deep information in unstructured data (such as voice and images). For example, a user saying "set the air conditioner temperature low" implies that they want to set the air conditioner temperature low, but traditional systems cannot automatically extract this semantic information. In addition, tags are mostly focused on demographic attributes (age, gender, region), lacking the depiction of dynamic characteristics such as behavioral patterns, interest migration, and emotional state, resulting in "flat" profiles that cannot support personalized services.
[0003] The update mechanism is outdated and cannot reflect real-time user changes. Traditional tags are mostly updated offline in batches, with cycles often on a daily or even weekly basis, which cannot meet the needs of rapid updates. For example, when faced with rapid shifts in user interests (such as from "warm air conditioning in spring" to "cooling in summer"), the system struggles to adjust in a timely manner.
[0004] Lacking predictive capabilities, traditional systems, based on static rules and labeling, can only describe the current situation and cannot predict future trends. For example, they cannot predict future user preferences based on user car usage behavior sequences, thus missing optimal outreach opportunities. In contrast, AI-driven systems can automatically discover hidden user needs such as "preference sensitivity and high reusability" through algorithms like clustering and deep learning, and predict car usage behavior preferences. These issues urgently need to be addressed. Summary of the Invention
[0005] The purpose of this invention is to provide an automatic expansion method and system for user profiles based on a multimodal large model, so as to solve the problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention provides the following technical solution: an automatic user profile expansion method based on a multimodal large model, characterized in that it includes:
[0007] Collect and preprocess multi-source user interaction data, extract multimodal features from the preprocessed multi-source data, and obtain a unified user feature vector;
[0008] Personalized new labels are generated from user feature vectors using a multimodal large model;
[0009] Construct a knowledge graph and associate user feature vectors with the knowledge graph;
[0010] The weights of tags in the original tag system are dynamically adjusted based on the new tags, and the original tag system is dynamically optimized to obtain expanded tag results in order to update the user profile.
[0011] Pre-defined business rules are used to validate user profiles. Validated user profiles are then applied to real-world business scenarios. A / B testing is used to compare business metrics under different tagging strategies to obtain test results.
[0012] Obtain user feedback, and adjust the parameters of the multimodal large model based on test results and user feedback, continuously iterating and optimizing.
[0013] Furthermore, multi-source data on user interactions is collected and preprocessed, including:
[0014] Differentiate the different data types from multiple sources, perform corresponding data cleaning for each data type, and normalize the cleaned data to eliminate dimensional differences between different data dimensions.
[0015] Furthermore, multimodal feature extraction is performed on the preprocessed multi-source data to obtain a unified user feature vector. The multi-source data includes text data, image data, and behavioral data. Semantic features are extracted from text data using the Transformer model, visual features are extracted from image data using the CNN or ViT model, and temporal features are extracted from behavioral data using the sequence model. The extracted multimodal features are then fused to form a unified user feature vector.
[0016] Furthermore, generating personalized new labels from user feature vectors using a multimodal large model also includes:
[0017] By leveraging the zero-shot learning capability of the multimodal large model, personalized new labels are generated from user feature vectors. A few-shot learning approach is used to input a small number of labeled examples for specific sub-domains, guiding the multimodal large model to generate new labels that fit business needs.
[0018] Furthermore, constructing a knowledge graph and associating user feature vectors with the knowledge graph also includes:
[0019] Construct a knowledge graph that integrates product classification knowledge, brand information knowledge, and user behavior preference knowledge within the industry.
[0020] By associating user feature vectors with knowledge graphs, and using knowledge reasoning to obtain potential user interest associations, we can uncover users' potential needs.
[0021] Furthermore, the weights of tags in the existing tagging system are dynamically adjusted based on the new tags to dynamically optimize the existing tagging system, obtain expanded tagging results, and update user profiles. This also includes:
[0022] By applying cosine similarity or BERTScore algorithms, the semantic similarity between the new label and the labels in the original label system is calculated to determine the hierarchical relationship and association strength between the labels.
[0023] Based on the frequency and importance of the new tags, dynamic weights are set for the tags in the original tag system to achieve dynamic optimization of the tag system.
[0024] Furthermore, the weights of tags in the existing tagging system are dynamically adjusted based on the new tags to dynamically optimize the existing tagging system, obtain expanded tagging results, and update user profiles. This also includes:
[0025] During the process of obtaining the expanded label results, the inference path and data basis of the multimodal large model are recorded, and an interpretable generation logic is provided for each new label.
[0026] Furthermore, the preset business rules for validating user profiles also include:
[0027] Determine whether the dynamically optimized tags conform to industry standards in the business rules and whether there are any logical contradictions. If they conform to industry standards and there are no logical contradictions, the verification is successful; otherwise, the verification is unsuccessful.
[0028] Furthermore, an evaluation mechanism for labels is set up based on the accuracy and business value of the labels. Labels that cannot be covered by business rules are evaluated through the evaluation mechanism to determine whether the labels are qualified.
[0029] On the other hand, an automatic user profile expansion system based on a multimodal large model is provided, which applies the automatic user profile expansion method based on a multimodal large model described above, including:
[0030] The preprocessing module collects and preprocesses multi-source user interaction data, performs multi-modal feature extraction on the preprocessed multi-source data, and obtains a unified user feature vector.
[0031] The new tag generation module generates personalized new tags from user feature vectors using a multimodal large model;
[0032] The knowledge graph construction module builds a knowledge graph and associates user feature vectors with the knowledge graph.
[0033] The tag system optimization module dynamically adjusts the weights of tags in the original tag system based on new tags, dynamically optimizes the original tag system, obtains expanded tag results, and updates user profiles.
[0034] The verification module uses preset business rules to verify user profiles, applies verified user profiles to actual business scenarios, and compares business metrics under different tagging strategies through A / B testing to obtain test results.
[0035] The iteration module gathers user feedback and adjusts the parameters of the multimodal large model based on test results and user feedback, continuously iterating and optimizing.
[0036] Compared with existing technologies, the beneficial effects of this invention are as follows: The multimodal large-scale model-driven automatic user profile expansion system overcomes the three major shortcomings of traditional tagging systems—single dimension, slow update, and weak predictive ability—by fusing unstructured data, dynamically generating semantic tags, and realizing intent reasoning and prediction. The multimodal large-scale model can uniformly parse unstructured information such as text, images, voice, video, and sensor data, greatly expanding the tag dimensions. The large-scale model supports real-time behavioral sequence modeling, enabling second-level dynamic tag updates. The Transformer-based temporal model can capture user behavior flows and automatically identify dynamic tags. The large-scale model can predict users' future needs through behavioral narrative modeling and knowledge graph fusion, enabling the profile system to move from passive response to proactive prediction. From modeling to interpretable user profile expansion, the large-scale model can not only generate tags but also explain the causes of tags using natural language, lowering the threshold for business use. By deeply understanding users' multimodal behavior, it automatically generates richer, more accurate, and more forward-looking user profiles. Attached Figure Description
[0037] Figure 1 This is a flowchart of the automatic user profile expansion method based on a multimodal large model in an embodiment of the present invention;
[0038] Figure 2 This is a connection block diagram of the automatic user profile expansion system based on a multimodal large model in an embodiment of the present invention. Detailed Implementation
[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0040] In the description of this invention, unless otherwise stated, "a plurality of" means two or more; the terms "upper," "lower," "left," "right," "inner," "outer," "front end," "rear end," "head," "tail," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0041] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0042] The core architecture of this invention adopts a four-layer architecture of "data layer - model layer - tag expansion engine layer - application layer", realizing a closed loop of the entire process from multi-source data access to intelligent tag expansion and then to business value output.
[0043] Please refer to the instruction manual appendix. Figure 1 As shown, this invention provides a technical solution: an automatic user profile expansion method based on a multimodal large model, comprising the following steps:
[0044] S102. Collect and preprocess multi-source data of user interaction, extract multi-modal features from the preprocessed multi-source data, and obtain a unified user feature vector.
[0045] The data layer integrates multi-source heterogeneous data, including user behavior tracking data (such as behavioral operation data), image data (such as facial expressions and eye movements), vehicle condition data (such as ignition, data, and location), and environmental data (such as geographical location and weather information). Through data cleaning, format standardization, and feature extraction, a standardized multimodal data pool is constructed to provide a high-quality data foundation for subsequent model training and scaling.
[0046] This includes collecting and preprocessing multi-source user interaction data, including:
[0047] Differentiate the different data types from multiple sources, perform corresponding data cleaning for each data type, and normalize the cleaned data to eliminate dimensional differences between different data dimensions.
[0048] In the above embodiments, targeted cleaning is performed on different types of data, such as removing noise information from text data, unifying the resolution and format of image data, and correcting outliers in behavioral data. At the same time, the data is normalized to eliminate the differences in units between different data dimensions.
[0049] In this process, multimodal feature extraction is performed on the preprocessed multi-source data to obtain a unified user feature vector. The multi-source data includes text data, image data, and behavioral data. Semantic features are extracted from text data using the Transformer model, visual features are extracted from image data using the CNN or ViT model, and temporal features are extracted from behavioral data using the sequence model. The extracted multimodal features are then fused to form a unified user feature vector.
[0050] In the above embodiments, dedicated models are used to extract features from each data type. Text data uses a Transformer model to extract semantic features, image data uses a CNN or ViT model to extract visual features, and behavioral data uses a sequence model to extract temporal features. The extracted multimodal features are then fused to form a unified user feature vector. For example, cockpit multimodal data fusion is preferred. Dedicated models are used to extract features from each data type. Text data uses a Transformer model to extract semantic features, image data uses a CNN or ViT model to extract visual features, and behavioral data uses a sequence model to extract temporal features. The extracted multimodal features are then fused with dedicated cockpit domain data.
[0051] S104. Generate personalized new labels from user feature vectors using a multimodal large model;
[0052] The model layer is based on a multimodal large model, which integrates domain fine-tuning technology. It selects basic large models with strong cross-modal understanding capabilities, such as GPT-4V and Gemini, and fine-tunes them using domain-specific labeled data for cockpit-specific business scenarios to enhance the model's ability to understand industry-specific semantics and user behavior. At the same time, a model evaluation and iteration mechanism is built to continuously optimize model parameters based on the scaling effect.
[0053] This includes generating personalized new labels from user feature vectors using a multimodal large model, as well as:
[0054] By leveraging the zero-shot learning capability of the multimodal large model, personalized new labels are generated from user feature vectors. A few-shot learning approach is used to input a small number of labeled examples for specific sub-domains, guiding the multimodal large model to generate new labels that fit business needs.
[0055] In the above embodiments, the generation of multiple samples and few-shot labels leverages the zero-shot learning capability of a large multimodal model, enabling the generation of new labels from user multimodal data without requiring a large amount of labeled data. For example, by inputting a user's voice history and emotional changes into the model, the model can automatically generate personalized labels such as "barbecue enthusiast" and "outdoor enthusiast." For specific niche areas, a few-shot learning approach is used, inputting a small number of labeled examples to guide the model in generating labels that better meet business needs.
[0056] S106. Construct a knowledge graph and associate user feature vectors with the knowledge graph;
[0057] The tag expansion engine layer, as the core processing module of the system, comprises three sub-engines: tag mining, tag association, and tag verification. The tag mining engine, based on the semantic reasoning capabilities of a multimodal large model, mines potential interests, needs, and preferences from multi-dimensional user data. The tag association engine, through knowledge graph technology, establishes semantic connections between new tags and the existing tag system, achieving organic expansion of the tag system. The tag verification engine, combining business rules and user feedback, verifies the accuracy and validity of the tag expansion results. The knowledge graph assists tag reasoning by constructing a domain knowledge graph, integrating industry knowledge such as service classifications, brand information, and user behavior preferences, effectively improving the accuracy of tag expansion and reducing the generation of false tags.
[0058] This includes constructing a knowledge graph and associating user feature vectors with the knowledge graph, as well as:
[0059] Construct a knowledge graph that integrates product classification knowledge, brand information knowledge, and user behavior preference knowledge within the industry.
[0060] By associating user feature vectors with knowledge graphs, and using knowledge reasoning to obtain potential user interest associations, we can uncover users' potential needs.
[0061] In the above embodiments, knowledge graphs assist in tag reasoning, constructing a domain knowledge graph that integrates industry-specific knowledge such as product classification, brand information, and user behavior preferences. During tag mining, user feature vectors are associated with the knowledge graph, and potential user interest associations are discovered through knowledge reasoning; for example, based on a user's music listening behavior, combined with the association relationship of "cabin ecology - music - music listened to while driving" in the knowledge graph, the user's potential music needs while driving are uncovered.
[0062] S108. Dynamically adjust the weight of tags in the original tag system based on the new tags, dynamically optimize the original tag system, obtain the expanded tag results, and update the user profile.
[0063] This includes dynamically adjusting the weights of tags in the existing tagging system based on the new tags, dynamically optimizing the existing tagging system to obtain expanded tagging results, and updating user profiles. It also includes:
[0064] By applying cosine similarity or BERTScore algorithms, the semantic similarity between the new label and the labels in the original label system is calculated to determine the hierarchical relationship and association strength between the labels.
[0065] Based on the frequency and importance of the new tags, dynamic weights are set for the tags in the original tag system to achieve dynamic optimization of the tag system.
[0066] In the above embodiments, algorithms such as cosine similarity and BERTScore are used to calculate the semantic similarity between newly added tags and tags in the original tag system, determining the hierarchical relationship and association strength between tags. Dynamic weights are assigned to tags based on the frequency and importance of user behavior changes. Tags corresponding to frequently occurring behaviors are given increased weights; tags that have not generated relevant behaviors for a long time are given decreased weights until they are eliminated, thus achieving dynamic optimization of the tag system.
[0067] This includes dynamically adjusting the weights of tags in the existing tagging system based on the new tags, dynamically optimizing the existing tagging system to obtain expanded tagging results, and updating user profiles. It also includes:
[0068] During the process of obtaining the expanded label results, the inference path and data basis of the multimodal large model are recorded, and an interpretable generation logic is provided for each new label.
[0069] In the above embodiments, for the sake of interpretability, the model reasoning path and data basis are recorded during the label expansion process, and an interpretable generation logic is provided for each new label to help business personnel understand the source of the labels and increase their trust in the label expansion results and their willingness to use them.
[0070] S110. Pre-set business rules to verify user profiles, apply verified user profiles to actual business scenarios, and compare business metrics under different labeling strategies through A / B testing to obtain test results.
[0071] At the application level, standardized interfaces are provided for different business scenarios, supporting business applications such as precision marketing, personalized recommendations, and product optimization.
[0072] The preset business rules for validating user profiles also include:
[0073] Determine whether the dynamically optimized tags conform to industry standards in the business rules and whether there are any logical contradictions. If they conform to industry standards and there are no logical contradictions, the verification is successful; otherwise, the verification is unsuccessful.
[0074] In the above embodiments, preset business rules perform preliminary verification of the label expansion results, such as whether the labels conform to industry standards and whether there are logical contradictions.
[0075] Among them, an evaluation mechanism for labels is set up based on the accuracy and business value of the labels. For labels that cannot be covered by business rules, the evaluation mechanism is used to evaluate whether the labels are qualified.
[0076] In the above embodiments, for complex tags that cannot be covered by the rules, a manual review mechanism is introduced, in which business experts evaluate the accuracy and business value of the tags.
[0077] S112. Obtain user feedback, and adjust the parameters of the multimodal large model based on test results and user feedback, and continuously iterate and optimize.
[0078] In the above embodiments, the expanded user profiles are applied to actual business scenarios, and business metrics such as conversion rate and click-through rate are compared under different tagging strategies through A / B testing. Based on the test results and user feedback, the model parameters and expansion rules are adjusted to achieve continuous iterative optimization of the system.
[0079] On the other hand, such as Figure 2 As shown, an automatic user profile expansion system based on a multimodal large model is provided, which applies the automatic user profile expansion method based on a multimodal large model described above, including:
[0080] The preprocessing module collects and preprocesses multi-source user interaction data, performs multi-modal feature extraction on the preprocessed multi-source data, and obtains a unified user feature vector.
[0081] The new tag generation module generates personalized new tags from user feature vectors using a multimodal large model;
[0082] The knowledge graph construction module builds a knowledge graph and associates user feature vectors with the knowledge graph.
[0083] The tag system optimization module dynamically adjusts the weights of tags in the original tag system based on new tags, dynamically optimizes the original tag system, obtains expanded tag results, and updates user profiles.
[0084] The verification module uses preset business rules to verify user profiles, applies verified user profiles to actual business scenarios, and compares business metrics under different tagging strategies through A / B testing to obtain test results.
[0085] The iteration module gathers user feedback and adjusts the parameters of the multimodal large model based on test results and user feedback, continuously iterating and optimizing.
[0086] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for automatic expansion of user profiles based on a multimodal large model, characterized in that, include: Collect and preprocess multi-source user interaction data, extract multimodal features from the preprocessed multi-source data, and obtain a unified user feature vector; Personalized new labels are generated from user feature vectors using a multimodal large model; Construct a knowledge graph and associate user feature vectors with the knowledge graph; The weights of tags in the original tag system are dynamically adjusted based on the new tags, and the original tag system is dynamically optimized to obtain expanded tag results in order to update the user profile. Pre-defined business rules are used to validate user profiles. Validated user profiles are then applied to real-world business scenarios. A / B testing is used to compare business metrics under different tagging strategies to obtain test results. Obtain user feedback, and adjust the parameters of the multimodal large model based on test results and user feedback, continuously iterating and optimizing.
2. The automatic user profile expansion method based on a multimodal large model according to claim 1, characterized in that, Collect and preprocess multi-source user interaction data, including: Differentiate the different data types from multiple sources, perform corresponding data cleaning for each data type, and normalize the cleaned data to eliminate dimensional differences between different data dimensions.
3. The automatic expansion method for user profiles based on a multimodal large model according to claim 1, characterized in that, Multimodal feature extraction is performed on preprocessed multi-source data to obtain a unified user feature vector. The multi-source data includes text data, image data, and behavioral data. Semantic features are extracted from text data using the Transformer model, visual features are extracted from image data using the CNN or ViT model, and temporal features are extracted from behavioral data using the sequence model. The extracted multimodal features are then fused to form a unified user feature vector.
4. The automatic user profile expansion method based on a multimodal large model according to claim 1, characterized in that, Generating personalized new labels from user feature vectors using a multimodal large model also includes: By leveraging the zero-shot learning capability of the multimodal large model, personalized new labels are generated from user feature vectors. A few-shot learning approach is used to input a small number of labeled examples for specific sub-domains, guiding the multimodal large model to generate new labels that fit business needs.
5. The automatic user profile expansion method based on a multimodal large model according to claim 1, characterized in that, Constructing a knowledge graph, associating user feature vectors with the knowledge graph, also includes: Construct a knowledge graph that integrates product classification knowledge, brand information knowledge, and user behavior preference knowledge within the industry. By associating user feature vectors with knowledge graphs, and using knowledge reasoning to obtain potential user interest associations, we can uncover users' potential needs.
6. The automatic user profile expansion method based on a multimodal large model according to claim 1, characterized in that, The weights of tags in the existing tagging system are dynamically adjusted based on the new tags to dynamically optimize the original tagging system, obtain expanded tagging results, and update user profiles. This also includes: By applying cosine similarity or BERTScore algorithms, the semantic similarity between the new label and the labels in the original label system is calculated to determine the hierarchical relationship and association strength between the labels. Based on the frequency and importance of the new tags, dynamic weights are set for the tags in the original tag system to achieve dynamic optimization of the tag system.
7. The automatic expansion method for user profiles based on a multimodal large model according to claim 1, characterized in that, The weights of tags in the existing tagging system are dynamically adjusted based on the new tags to dynamically optimize the original tagging system, obtain expanded tagging results, and update user profiles. This also includes: During the process of obtaining the expanded label results, the inference path and data basis of the multimodal large model are recorded, and an interpretable generation logic is provided for each new label.
8. The automatic expansion method for user profiles based on a multimodal large model according to claim 1, characterized in that, Pre-defined business rules for validating user profiles also include: Determine whether the dynamically optimized tags conform to industry standards in the business rules and whether there are any logical contradictions. If they conform to industry standards and there are no logical contradictions, the verification is successful; otherwise, the verification is unsuccessful.
9. The automatic expansion method for user profiles based on a multimodal large model according to claim 8, characterized in that, include: An evaluation mechanism for labels is set up based on the accuracy and business value of the labels. Labels that cannot be covered by business rules are evaluated through the evaluation mechanism to determine whether the labels are qualified.
10. An automatic user profile expansion system based on a multimodal large model, employing the automatic user profile expansion method based on a multimodal large model as described in any one of claims 1 to 9, characterized in that, include: The preprocessing module collects and preprocesses multi-source user interaction data, performs multi-modal feature extraction on the preprocessed multi-source data, and obtains a unified user feature vector. The new tag generation module generates personalized new tags from user feature vectors using a multimodal large model; The knowledge graph construction module builds a knowledge graph and associates user feature vectors with the knowledge graph. The tag system optimization module dynamically adjusts the weights of tags in the original tag system based on new tags, dynamically optimizes the original tag system, obtains expanded tag results, and updates user profiles. The verification module uses preset business rules to verify user profiles, applies verified user profiles to actual business scenarios, and compares business metrics under different tagging strategies through A / B testing to obtain test results. The iteration module gathers user feedback and adjusts the parameters of the multimodal large model based on test results and user feedback, continuously iterating and optimizing.