Object portrait label set generation method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses an object portrait label set generation method, device and equipment and a medium, and the method comprises the steps: collecting and integrating multi-source data, generating a multi-modal data set, and cleaning and enhancing the data set; extracting a portrait label set based on the historical structured information of the object, obtaining a multi-modal cue word set, and constructing a unified representation sequence; inputting the unified representation sequence into a multi-modal model, obtaining fusion representation and extracting a derivative factor candidate set; and performing consistency verification on the cross-modal data to obtain a consistency factor set, and finally fusing the consistency factor set with the structured portrait label set to generate a unified portrait system and output an object portrait label set. According to the method, the accurate and comprehensive customer portrait is generated by combining multi-modal data and cross-modal consistency verification, the portrait accuracy is improved, and support is provided for accurate marketing and personalized service.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an object portrait label set generation method and device, equipment and medium. BACKGROUND

[0002] In the field of financial technology business, customer portrait technology has been widely used in precision marketing, risk control and customer service management. Traditional customer portraits rely mainly on structured data, such as customer gender, age, transaction amount and purchase frequency. This kind of portrait method can support basic customer classification and group analysis at the initial stage, but it has obvious shortcomings in complex business environment. First, single reliance on structured data often fails to reflect the dynamic needs of customers in the actual interaction process, which easily leads to one-sided interpretation of customer behavior. Second, with the emergence of diversified business scenarios such as mobile finance and intelligent customer service, customer expression preferences and potential intentions are often contained in unstructured data such as voice interaction, image clicks and text consultation. However, traditional methods cannot effectively integrate these multi-modal data, resulting in insufficient comprehensiveness and authenticity of the portrait. Third, the existing customer portrait system lacks a consistent verification mechanism for cross-modal factors, making it difficult for features from different data sources to form a unified portrait system, which limits its application depth in precision marketing and differentiated services.

[0003] In the field of medical and health business, patient portraits have gradually become an important tool for assisting diagnosis and treatment, health care service recommendation and health management. Existing medical and health portrait methods also mainly rely on structured information, such as basic demographic data, historical medical records and medical expenses. Although these data can provide basic references for hospitals or health management agencies, there are still some shortcomings. First, structured data cannot fully reflect the behavior patterns and emotional states of patients in consultation, rehabilitation or service interaction, making it difficult to support more personalized health service design. Second, there is a large amount of unstructured data in medical and health business, such as doctor-patient voice communication records, online consultation texts, rehabilitation behavior logs and medical images. These information contains rich patient intentions, emotions and behavior characteristics, but existing methods do not make full use of them, often causing service design to be out of touch. Finally, there is a lack of verification of feature consistency between different data modalities, which easily leads to conflicts or bias accumulation of different source data, thus reducing the credibility and practicality of health portraits. SUMMARY

[0004] The main purpose of the present application is to provide an object portrait label set generation method, device, equipment and storage medium, aiming to solve the technical problem that traditional user portrait methods excessively rely on structured data, lack effective integration of multi-modal data and cross-modal consistency verification, resulting in one-sided and lack of unified portrait results.

[0005] To achieve the above object, the present application provides an object image label set generation method, comprising:

[0006] Collect and integrate multi-source data to generate a multi-modal data set, perform cleaning and enhancement operations on the multi-modal data set to generate a data preprocessing result;

[0007] Extract a structured image label set based on object historical structured information;

[0008] Obtain a multi-modal prompt word set, and construct a unified representation sequence based on the data preprocessing result and the multi-modal prompt word set;

[0009] Input the unified representation sequence into a multi-modal model to obtain a fusion representation, and extract a derived factor candidate set from the fusion representation;

[0010] Perform cross-modal consistency verification on cross-modal data in the derived factor candidate set to obtain a consistent factor set;

[0011] Fuse the consistent factor set and the structured image label set to generate a unified image system, and output an object image label set in the unified image system.

[0012] Further, to achieve the above object, the present application provides an object image label set generation device, comprising:

[0013] A data acquisition and preprocessing module is configured to collect and integrate multi-source data to generate a multi-modal data set, perform cleaning and enhancement operations on the multi-modal data set to generate a data preprocessing result;

[0014] A label extraction module is configured to extract a structured image label set based on object historical structured information;

[0015] A prompt word construction module is configured to obtain a multi-modal prompt word set, and construct a unified representation sequence based on the data preprocessing result and the multi-modal prompt word set;

[0016] A factor extraction module is configured to input the unified representation sequence into a multi-modal model to obtain a fusion representation, and extract a derived factor candidate set from the fusion representation;

[0017] A consistency verification module is configured to perform cross-modal consistency verification on cross-modal data in the derived factor candidate set to obtain a consistent factor set;

[0018] An image fusion output module is configured to fuse the consistent factor set and the structured image label set to generate a unified image system, and output an object image label set in the unified image system.

[0019] Further, in order to achieve the above object, the present application also provides a computer device, comprising a memory, a processor, and an object portrait label set generation program stored in the memory and executable on the processor, which, when executed by the processor, implements the steps of the object portrait label set generation method as described above.

[0020] Further, in order to achieve the above object, the present application also provides a computer readable storage medium, which stores an object portrait label set generation program, which, when executed by a processor, implements the steps of the object portrait label set generation method as described above.

[0021] Beneficial effects: The present application relates to the field of artificial intelligence technology, and can be applied to business scenarios such as financial technology and medical health, and discloses an object portrait label set generation method, device, equipment and medium, which comprises the following steps: collecting and integrating multi-source data, generating a multi-modal data set, and performing cleaning and enhancement operations on the data set; extracting a structured portrait label set based on object historical structured information, obtaining a multi-modal prompt word set, and constructing a unified representation sequence; inputting the unified representation sequence into a multi-modal model to obtain fusion representation and extract a derived factor candidate set therefrom; performing consistency verification on cross-modal data in the derived factor candidate set to obtain a consistent factor set; and finally, fusing the consistent factor set with the structured portrait label set to generate a unified portrait system and output an object portrait label set. The present application generates a more accurate and comprehensive customer portrait by combining multi-modal data and cross-modal consistency verification, can avoid the deviation caused by a single data source in traditional methods, improves the accuracy and comprehensiveness of the customer portrait, and thus provides stronger support for precise marketing and personalized services. BRIEF DESCRIPTION OF DRAWINGS

[0022] The present application will be further described below with reference to the accompanying drawings and embodiments, in which:

[0023] Figure 1 An application environment schematic diagram of the object portrait label set generation method in an embodiment of the present application;

[0024] Figure 2 A flowchart of an embodiment of the object portrait label set generation method of the present application;

[0025] Figure 3 A functional module schematic diagram of a preferred embodiment of the object portrait label set generation device of the present application;

[0026] Figure 4 A structure schematic diagram of a computer device in an embodiment of the present application;

[0027] Figure 5This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0028] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0029] The object profile tag set generation method provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can collect and integrate multi-source data through the client to generate a multimodal dataset, and perform cleaning and enhancement operations on the dataset; extract a structured profile tag set based on the object's historical structured information, obtain a multimodal prompt word set, and construct a unified representation sequence; input the unified representation sequence into the multimodal model to obtain a fused representation and extract a candidate set of derived factors; perform consistency verification on the cross-modal data in the candidate set of derived factors to obtain a consistent factor set; finally, merge the consistent factor set with the structured profile tag set to generate a unified profile system and output the object profile tag set. This invention, by combining multimodal data and cross-modal consistency verification, generates a more accurate and comprehensive customer profile, avoiding the bias caused by a single data source in traditional methods, improving the accuracy and comprehensiveness of the customer profile, and thus providing stronger support for precision marketing and personalized services. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0030] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the object profile tag set generation method provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0031] like Figure 2 As shown, the object profile tag set generation method proposed in this invention includes the following steps:

[0032] S10, Collect and integrate multi-source data to generate a multimodal data set, perform cleaning and enhancement operations on the multimodal data set, and generate data preprocessing results;

[0033] In this embodiment, a fundamental step in generating a customer profile is collecting and integrating multi-source data to create a multimodal dataset, followed by cleaning and enhancement operations on this dataset. This process aims to ensure the final customer profile is highly accurate and representative. To achieve this, multiple data sources are first collected and integrated, including text, voice, images, and behavioral logs. These data are stored in their respective storage spaces; for example, text data is stored in a text storage area, voice data in a voice storage area, and image data in an image storage area. The data integration process involves extracting data from different storage areas and generating a multimodal dataset through data formatting and alignment. The key technology in this process lies in integrating various data sources, ensuring that different types of data can be processed in a unified format, guaranteeing that subsequent analysis is not affected by differences in data types.

[0034] In the data preprocessing stage, text data in the multimodal dataset is processed first. Common operations include missing value imputation, outlier detection, and noise removal. These steps ensure the reasonableness of missing data by defining reasonable data imputation rules (such as mean imputation, KNN imputation, etc.), while using anomaly detection algorithms (such as standard deviation methods based on statistical analysis or outlier detection based on machine learning) to remove outliers that may affect model training. Speech data preprocessing focuses on noise reduction, which is typically performed using spectral analysis-based filters or deep learning models to remove background noise and improve the signal quality of the speech data. Image data enhancement includes various operations such as image rotation, cropping, flipping, and color remapping, aiming to increase sample diversity and enhance the model's generalization ability. These enhancement techniques help address noise and inconsistencies in image data.

[0035] These processed data are combined to form the data preprocessing result, which serves as the foundation for subsequent customer profiling. The data preprocessing result is a refined version of the multimodal dataset; after cleaning and enhancement, it provides high-quality data support for the extraction of customer profiling derivative factors.

[0036] Different data storage and processing methods can be adopted in different implementation environments. In the financial sector, data storage typically uses efficient relational databases. Text data may come from bank customer communication records, emails, social media platforms, etc., while voice data comes from call records at customer service centers, and image data may come from customer-uploaded ID cards, signed documents, etc. Noise reduction for voice data may be achieved using deep learning models, and image data enhancement can be combined with generative adversarial networks (GANs).

[0037] In the healthcare field, data sources may include patients' medical records, doctors' voice suggestions, and images uploaded by patients (such as X-rays, MRI scans, etc.). Preprocessing of text data may involve standardizing medical record data, noise reduction of voice data may utilize noise suppression techniques specifically designed for the medical field, and image enhancement relies on specialized algorithms in the field of medical image processing.

[0038] The flexibility and adaptability of the data preprocessing stage enable this method to be implemented in various application scenarios. By rationally selecting different data storage, cleaning, and enhancement strategies, data quality can be effectively improved, ensuring the accuracy and precision of the final customer profile.

[0039] This embodiment utilizes refined cleaning and enhancement of multi-source data to effectively improve the quality of multimodal datasets, providing a solid foundation for subsequent extraction of derived factors and construction of a customer profiling system. The data preprocessing not only eliminates noisy data but also improves data consistency and reliability, thereby ensuring the accuracy and comprehensiveness of customer profiles. This processing method not only reduces the impact of data bias in traditional customer profiling but also enhances the model's adaptability and generalization ability when facing complex and diverse data.

[0040] S20, Extract a set of structured profile tags based on the object's historical structured information;

[0041] In this embodiment, extracting a set of structured profile tags from the object's historical structured information is a crucial step in building a customer profile. The main goal of this process is to extract tags that help accurately characterize customer traits from structured data such as historical behavioral data, transaction records, and basic information. First, it is necessary to collect information including historical transaction data, customer behavior logs, and basic attribute data. Transaction data generally includes the customer's purchase history, purchase frequency, and amount; behavior log data includes records of customer interactions with the platform, such as login frequency, visited pages, and search history; basic attribute data includes basic personal information such as gender, age, and occupation.

[0042] The specific implementation process for extracting a structured profile tag set first involves identifying relevant tags from historical transaction data, such as purchase frequency tags and transaction amount tags. Purchase frequency tags are generated by calculating the number of purchases a customer makes within a certain time frame, reflecting customer activity and loyalty. Transaction amount tags are generated by statistically analyzing the total amount spent by customers in historical transactions, reflecting their spending power. Secondly, relevant tags also need to be extracted from behavioral log data. For example, interaction activity tags are generated based on the frequency and duration of customer interactions with the platform, while access frequency tags are generated based on the number of times a customer visits the platform. These behavioral tags reflect the customer's activity level and engagement on the platform. Preference migration tags can be generated by analyzing historical trends in customer preferences; these tags help businesses understand changes in customer needs and preferences.

[0043] Among the basic customer attribute data, gender and age are the most fundamental and commonly used tags. Gender tags are typically extracted from the customer's basic information, while age tags are calculated and labeled using the difference between the customer's date of birth and the current time. These tags help to categorize customers simply and provide data support for further customer segmentation and targeted marketing.

[0044] These extracted set of structured tags include purchase frequency tags, transaction amount tags, interaction activity tags, visit frequency tags, preference migration tags, gender attribute tags, and age attribute tags. These are the foundational data for building multi-dimensional customer profiles.

[0045] In the financial sector, customer transaction records can be obtained through channels such as bank systems and credit card spending records. Purchase frequency and transaction amounts can be calculated using the bank's back-end transaction system. Behavioral log data comes from customer interactions with the platform, such as online banking records and customer spending history. Basic attribute data, such as gender and age, typically comes from the identity information provided by the customer when conducting business at the bank.

[0046] In the healthcare field, customer behavioral data can include patient activity records on hospitals or health management platforms, such as appointment frequency, medical history, and medication purchase records. Purchase frequency and transaction amount tags can be generated from patients' medication purchase records, while interaction activity tags are calculated based on patients' interactions with the healthcare platform. Gender and age tags can be directly obtained from patients' basic information.

[0047] Implementation methods in different fields can be adjusted according to the specific source of the data. In practice, data acquisition can rely on standardized interfaces or databases within the industry, and extraction and tag generation must be completed while ensuring data accuracy.

[0048] This embodiment extracts a set of structured profile tags from customers' historical structured information, effectively reflecting their behavioral characteristics and basic attributes to generate refined customer profiles. These tag sets not only help understand customers' consumption patterns and activity levels but also provide strong data support for subsequent steps such as personalized recommendations and precision marketing. Accurate tag extraction effectively reduces biases introduced by single data sources, enhancing the comprehensiveness and accuracy of customer profiles. This method provides enterprises with more precise customer insights, optimizes user experience, and improves business operational efficiency.

[0049] S30, Obtain a multimodal prompt word set, and construct a unified representation sequence based on the data preprocessing result and the multimodal prompt word set;

[0050] In this embodiment, obtaining a set of multimodal cue words and constructing a unified representation sequence based on the data preprocessing results and the cue word set are crucial steps in building a multimodal customer profile. The goal of this process is to transform multi-source, multi-type data into a unified representation, thereby providing consistent and efficient input for subsequent multimodal model processing.

[0051] First, the acquisition of a multimodal cue word set is to help construct a unified representation sequence. These cue words are generated based on the characteristics of different modalities, typically including feature information from text, speech, and image data. Text cue words can be generated based on the topic or semantic features of text data, such as extracting keywords or thematic terms from customers' historical interaction records. Speech cue words are generated based on the emotional features, tone, and speech rate of speech data, capable of capturing the emotional changes of customers in the conversation. Image cue words are generated based on the features of image content, such as the category of items in the image, visual style, or image color.

[0052] When acquiring a multimodal cue word set, the first step is to generate cue words related to the features of each data modality. For text data, cue words can be generated using natural language processing techniques such as TF-IDF, Word2Vec, or BERT models, which help extract key information from the text. For speech data, cue words can be generated using speech recognition systems, converting the speech signal into text for further analysis of emotional features and extraction of relevant emotional keywords. Cue words for image data can be generated using computer vision algorithms, such as using convolutional neural networks (CNNs) to extract visual features like object categories, colors, and shapes from images as part of the cue words.

[0053] After generating the multimodal cue word set, a unified representation sequence needs to be constructed by combining the data preprocessing results (i.e., the data after cleaning and enhancement operations on the multimodal data). Preprocessing results typically include text data standardization, speech data denoising, and image data enhancement. These preprocessing operations ensure data consistency, integrity, and high quality for subsequent processing. Based on the combination of this preprocessed data and the multimodal cue word set, the construction of the unified representation sequence can be achieved by fusing features from text, speech, and images. For example, token sequences from text data, recognition tokens from speech data, and visual tokens from image data can all be concatenated or fused with their corresponding cue word sets to form a unified representation sequence. This ensures that information from different modalities is effectively integrated and passed on to subsequent models.

[0054] In the financial sector, acquiring a multimodal cue word set can be achieved through the following steps: First, extract text cue words from customer transaction records, which can be done by analyzing text data such as customer bank transaction descriptions and SMS notifications. For voice data, relevant voice cue words can be extracted from call center voice recordings or dialogue content of intelligent customer service systems, such as changes in customer emotions (pleasure, anxiety, etc.). Image data cue words can be extracted by analyzing visual content such as customer-uploaded bank card information and financial product images, extracting features such as product category and color. Finally, these text, voice, and image cue words are combined with corresponding preprocessed data to form a unified representation sequence, which serves as the basis for subsequent multimodal model input.

[0055] In the healthcare field, the process of obtaining a multimodal cue word set is similar: text cue words are extracted from patients' medical records, voice cue words are extracted from doctors' and patients' voice conversations, and image cue words are derived from the analysis of imaging examination results such as X-rays and CT images. The generated cue word set needs to be combined with cleaned and enhanced medical data, such as patients' medical histories, diagnoses, and medication records, to ultimately form a unified representation sequence for subsequent model analysis.

[0056] This embodiment constructs a unified representation sequence by acquiring and combining a multimodal cue word set with data preprocessing results. This effectively integrates data from different modalities, reduces bias caused by single-modal data, and enhances the multidimensional representation of the data. The multimodal representation sequence provides high-quality input for subsequent large-scale machine learning models, improving the model's performance in multimodal data fusion and resulting in more accurate and comprehensive customer profiles. This method enables more detailed customer profiling, helping businesses conduct customer analysis and precision marketing from different dimensions.

[0057] S40, input the unified representation sequence into the multimodal model to obtain a fusion representation, and extract a candidate set of derived factors from the fusion representation;

[0058] In this embodiment, inputting the unified representation sequence into the multimodal model to obtain a fused representation is a crucial step in constructing a multimodal customer profile. The purpose of this process is to deeply fuse data from different modalities (such as text, speech, and images) through a large model to extract derivative factors that are significant for the customer profile. This process first relies on the unified representation sequence generated in the preceding steps, which is a high-dimensional representation obtained through processing multimodal data and combining it with prompts.

[0059] In this process, the unified representation sequence is input into a multimodal model, which is typically a deep learning-based model, such as the Transformer architecture, particularly well-suited for processing and fusing data from different modalities. The core function of the multimodal model is to process the unified representation sequence, extract deep-level features from each modality, learn the relationships between these features through the model's training process, and ultimately output a fused representation.

[0060] Fusion representation can be viewed as a multi-dimensional representation that incorporates not only the semantic information of text data but also the emotional information of speech data and the visual features of image data. Information from each modality is abstracted and integrated through different levels of the larger model to form a multi-level, multi-dimensional comprehensive feature representation.

[0061] Once the fused representation is obtained, the next task is to extract a candidate set of derived factors. This candidate set refers to the key factors relevant to the customer profile extracted from the fused representation. These factors may include customer sentiment, purchase intent, product preferences, and behavioral patterns. This extraction process typically involves feature selection or dimensionality reduction of the fused representation to ensure that factors with practical significance and strong relevance to predicting customer behavior are extracted. These candidate sets of derived factors will provide data support for subsequent customer profile construction.

[0062] In the financial sector, the process of obtaining fused representations may involve the following steps: First, a unified representation sequence generated from customer transaction records, voice customer service recordings, and image data of customers browsing product pages is input into a multimodal model for training and fusion. Through this large model, financial institutions can extract multiple dimensions of features related to customer financial behavior, such as customer spending frequency, emotional attitudes (e.g., interest in a particular financial product), and preferences for a specific type of financial service. By extracting a candidate set of derived factors, financial institutions can further understand customer consumption psychology and needs, thereby providing personalized financial product recommendations or optimizing customer relationship management.

[0063] In the healthcare field, patient history, voice consultation records, and medical imaging data can be fused using a unified representation sequence. After being input into a large model, the healthcare system can extract key information from the patient's multimodal data, such as emotional state, disease prediction, and the patient's medical needs. For example, a doctor's voice conversation with a patient may reveal the patient's level of anxiety; this emotional information is crucial for patient health management. By extracting a candidate set of derived factors, the healthcare system can obtain a more accurate patient profile, thereby providing personalized treatment plans.

[0064] This embodiment significantly improves the multi-dimensional accuracy of customer profiling by inputting a unified representation sequence into a multimodal model and obtaining a fused representation, from which a candidate set of derived factors is extracted. The fusion of multimodal data ensures a comprehensive understanding of customer or patient behavior, needs, and preferences, thereby avoiding the bias of a single data source and more accurately reflecting the true state of the customer or patient.

[0065] S50, Perform cross-modal consistency verification on the cross-modal data in the candidate set of derived factors to obtain a consistent factor set;

[0066] In this embodiment, the generation of the candidate set of derived factors during multimodal data mining involves processing and fusing features from different data modalities (such as text, speech, and images) to form a comprehensive customer or object profile. Cross-modal consistency verification is a crucial step in this process, aiming to ensure that features extracted from different modalities (such as text and images, speech and text, etc.) are semantically consistent, thereby improving the accuracy and reliability of the final customer profile.

[0067] The essence of cross-modal consistency verification is to check whether the factors extracted from different modalities support each other and are comparable at the same level. For example, the sentiment factors in text data and the sentiment factors in voice data should express consistent emotional states, and the user's points of interest in image data should match the descriptions in text data. This verification process calculates the similarity between modalities to determine whether the features extracted from different modalities can jointly reflect the true state of the customer or object.

[0068] This process typically relies on pre-trained large multimodal models that learn the mapping relationships between different modalities. Cross-modal consistency verification calculates the similarity between factors in each modality and factors in other modalities, setting a consistency threshold. When the similarity between factors from two or more modalities exceeds this threshold, these factors are considered consistent, retained, and merged into a consistent factor set. This consistency verification not only improves the accuracy of factors but also reduces data distortion caused by biases between different modalities.

[0069] In practical implementation, firstly, deep learning models (such as Transformer or BERT) are used to extract features from multimodal data such as text, speech, and images, generating corresponding representation vectors. Then, the similarity between representation vectors of different modalities is calculated, using methods such as cosine similarity and Euclidean distance for measurement. Finally, based on a set threshold, it is determined which factors need to be retained and added to the consistency factor set. This process ensures the semantic consistency of derived factors extracted from different modalities, thereby improving the reliability of the overall customer profile.

[0070] In the financial sector, cross-modal consistency verification can be applied to multi-dimensional customer data (such as financial transaction data, voice customer service data, and web browsing history). For example, by analyzing the emotional information in a customer's voice and text conversations, it's possible to determine their interest or sentiment towards certain financial products. Then, by examining the product images and text descriptions viewed by the customer on a webpage, it's possible to check whether their interests and needs align with the sentiment analysis. Through consistency verification, the system can ensure that the customer intentions reflected by this data from different modalities are consistent, thereby avoiding erroneous predictions due to biases from a single data source. Ultimately, these consistent factors will be incorporated into a consistency factor set for subsequent personalized financial product recommendations.

[0071] In the healthcare field, cross-modal consistency verification also plays a crucial role. For example, patient medical records, voice consultation data, and medical imaging data can provide different levels of health information. Emotional factors extracted from voice and text data (such as a patient's anxiety level) should be consistent with the health status reflected in medical imaging (such as heart rate changes). Through consistency verification, the system can ensure that a patient's emotional state matches their health data, avoiding inaccurate medical service recommendations due to incorrect emotional state judgments. The unified and consistent factors obtained through this process will improve the accuracy and personalization of disease diagnosis and treatment plans.

[0072] This embodiment ensures semantic and contextual consistency of factors extracted from different modalities through cross-modal consistency verification. This process significantly improves the accuracy and reliability of derived factors extracted from multimodal data, reducing erroneous judgments that may result from biases in a single data source. Ultimately, the generated consistent factor set provides a more comprehensive and accurate customer or object profile, offering a more reliable foundation for subsequent applications such as precision marketing, personalized services, and healthcare management.

[0073] S60, merge the set of consistent factors with the set of structured profile tags to generate a unified profile system, and output the set of object profile tags in the unified profile system.

[0074] In this embodiment, generating a unified profile system is a crucial step in the multimodal data mining process. The core objective of this step is to fuse the set of consistency factors obtained through cross-modal consistency verification with the structured profile tag set, thereby constructing a more comprehensive and accurate customer or object profile system. The fusion process combines data from two different sources (i.e., cross-modal data and structured tags) to generate a comprehensive, three-dimensional, multi-faceted customer profile, providing richer foundational data for subsequent applications such as precision marketing and personalized recommendations.

[0075] The consistency factor set contains factors from different data modalities (such as text, voice, and images), representing the state and needs of customers or objects in different contexts. The structured profile tag set contains standardized tags extracted from historical data (such as transaction records, access records, and basic attributes). These tags are often extracted using traditional structured data processing methods, reflecting the customer's behavior and characteristics over a past period.

[0076] In this step, the fusion process goes beyond simply merging the consistent factor set and the structured tag set. It involves a weighted integration of the two, ensuring they support, complement, and enhance each other in the final unified profile system. Through this process, the system generates a comprehensive profile system where the object profile tag set includes both structured tags based on historical behavioral data and derived factors extracted from multimodal data. This provides a more detailed, accurate, and dynamic customer or object profile.

[0077] The specific implementation of this process typically relies on algorithmic models (such as weighted summation, weighted average, etc.) to calculate the importance of various factors and labels, thereby achieving a reasonable fusion. Technically, the first step is to normalize different types of factors and labels to ensure their numerical comparability. Next, weighted fusion is performed by combining their respective weights. Finally, the generated unified profile system will contain multi-dimensional customer information, enabling it to play a greater role in business scenarios such as personalized recommendations and precision marketing.

[0078] In the financial sector, the generation of a unified customer profile system can be used to integrate multi-dimensional data such as customer behavior, emotions, and preferences. For example, during customer service, a customer expresses interest in a certain type of financial product via telephone customer service (extracted through voice emotion factors), and their financial transaction records show a high transaction amount and frequency. By fusing these two types of data, a comprehensive customer profile can be generated. This profile not only reflects the customer's behavior (such as purchase frequency and transaction amount) but also includes the customer's emotional preferences (such as emotional bias towards a certain type of product). The final output set of customer profile tags can be provided to financial institutions for accurately recommending financial products that meet customer needs.

[0079] In the healthcare field, a unified patient profile system can help hospitals and health management institutions gain a more comprehensive understanding of patients' health status. For example, a patient's imaging results, historical medical records, and emotional analysis (anxiety, depression, etc.) during voice consultations can be integrated through this process. By comprehensively considering the patient's imaging data and emotional state, a more accurate health profile is generated, providing doctors with more multidimensional information when developing treatment plans. This integration not only improves the personalization of healthcare services but also helps healthcare institutions provide more comprehensive data support when managing patient health.

[0080] This embodiment, by fusing a consistent set of factors and a structured profile tag set, generates a unified profile system that provides more comprehensive and accurate customer or target information than traditional profiles. This fusion not only improves the accuracy of the profile but also reduces errors that may arise from biases caused by a single data source. The final output target profile tag set includes tags based on traditional structured data as well as deep features provided by multimodal data, providing more detailed and dynamic foundational data support for subsequent applications such as precision marketing, personalized services, and healthcare management.

[0081] This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a method, apparatus, device, and medium for generating object profile tag sets, comprising: collecting and integrating multi-source data to generate a multimodal data set, and performing cleaning and enhancement operations on the data set; extracting a structured profile tag set based on the object's historical structured information, obtaining a multimodal cue word set, and constructing a unified representation sequence; inputting the unified representation sequence into a multimodal model to obtain a fused representation and extracting a candidate set of derived factors; performing consistency verification on cross-modal data in the candidate set of derived factors to obtain a consistent factor set; and finally, merging the consistent factor set with the structured profile tag set to generate a unified profile system and outputting the object profile tag set. This invention, by combining multimodal data and cross-modal consistency verification, generates more accurate and comprehensive customer profiles, avoiding the bias caused by a single data source in traditional methods, improving the accuracy and comprehensiveness of customer profiles, and thus providing stronger support for precision marketing and personalized services.

[0082] In one embodiment, step S10 includes:

[0083] S101, Collect text data, voice data and image data, store the text data in the text storage area, store the voice data in the voice storage area, and store the image data in the image storage area;

[0084] S102, extract data from the text storage area, voice storage area and image storage area, integrate the extracted data, and generate a multimodal data set;

[0085] S103, perform missing value imputation, outlier detection and cleaning operations on the text data in the multimodal data set;

[0086] S104, Perform noise reduction processing on the speech data in the multimodal data set;

[0087] S105, perform enhancement processing on the image data in the multimodal data set;

[0088] S106 integrates text data after missing value imputation, outlier detection and cleaning, speech data after noise reduction, and image data after enhancement to generate data preprocessing results.

[0089] In this embodiment, text data, voice data, and image data are collected to establish a unified access protocol and identity mapping for multi-source heterogeneous entry points. Text data comes from online interaction records, business document fields, Q&A records, etc., and is uniformly encoded in UTF-8 while preserving the original line breaks and timestamps. Voice data comes from call recordings and voicemail messages, with unified sampling rate and bit depth, and supplemented with metadata such as channel number and duration. Image data comes from images uploaded by the terminal and screenshots of business traces, with unified color space and resolution, and supplemented with shooting time, device type, and aspect ratio. All three types of data access simultaneously generate globally unique object identifiers and session identifiers for subsequent cross-modal alignment and deduplication, preventing the same event from generating fragmented records at different entry points.

[0090] The text, audio, and image storage areas organize raw data and metadata using a combination of physical partitioning and logical naming. The text storage area employs a two-tiered structure of columnar tables or object buckets, with the partition key being the date and object identifier, recording the text, source, language, encoding checksum, and more. The audio storage area uses a binding model of object buckets and index tables. Each bucket stores the raw audio and fingerprint hash, while the index table maintains the sampling rate, bit rate, channels, recording endpoint timestamps, and silence percentage. The image storage area uses a hybrid structure of object buckets and feature indexes, storing the raw images and thumbnails. The index side records exposure estimates, sharpness scores, noise estimates, and EXIF ​​features. All three types of storage areas record data integrity check values ​​and batch numbers, forming a traceable version sequence to meet subsequent quality backtracking and reprocessing needs.

[0091] When extracting and integrating data from text, voice, and image storage areas to generate a multimodal dataset, the aggregation unit is constructed using object identifiers and time-aligned windows as the primary keys. First, cross-channel records of the same object are aggregated based on object identifiers. Then, semantically related fragments are aligned using sliding time windows or session boundaries to form composite records containing text, voice, and image sets. During alignment, duplicate detection and conflict resolution are performed. Duplicate detection is determined by hash similarity and time overlap, while conflict resolution follows priority rules based on source credibility, time proximity, and content completeness. After integration, a structured index of the multimodal dataset is output. Index items include object identifiers, window start and end times, modality coverage matrix, and quality score, ensuring that subsequent processing can select appropriate paths based on coverage.

[0092] When performing missing value imputation, outlier detection, and cleaning operations on text data in a multimodal dataset, a two-layer check is first performed at the field level and semantic level. Field-level checks identify empty strings, blank spaces, illegal encoding, and truncation markers; semantic-level checks identify extremely short text, repetitive template text, and garbled characters. For missing value imputation, enumerated fields are filled with valid values; numerical fields are imputed based on object historical quantiles; and time fields are imputed using adjacent records with imputation markers recorded. Outlier detection employs a hybrid rule-based and statistical strategy. The rule-based approach includes length boundaries, blacklist regularization, and HTML remnants; the statistical approach includes character distribution deviation and word frequency confidence intervals. Cleaning operations sequentially perform HTML and tag removal, control character cleanup, full-width and half-width character standardization, noise symbol compression, language unification, and spelling standardization, and de-identify or hash sensitive entities. All replacements and deletions resulting from cleaning are recorded in the change log to support auditing and rollback.

[0093] When performing noise reduction on speech data in a multimodal dataset, resampling and amplitude normalization are first performed, followed by speech activity detection to segment silences and speech segments. For each speech segment, spectral subtraction, Wiener filtering, or a time-frequency masking network is selected for noise reduction based on the noise type. In mixed noise scenarios, adaptive noise profile estimation is added. To suppress echo and reverberation, the room impulse response approximation is calculated and déreverberation is performed, followed by noisy energy thresholding for click noise and plosives. After noise reduction, phase consistency correction and loudness normalization are performed, and the audio and quality metrics are output, including signal-to-noise ratio improvement, short-term intelligibility, and residual noise scores. If the speech contains double-talk segments, overlap detection is used to separate the tracks and process them separately, and finally they are synthesized and aligned to the conversation timeline.

[0094] When performing enhancement processing on image data in a multimodal dataset, orientation correction and aspect ratio normalization are performed first, followed by evaluation of exposure, contrast, and sharpness metrics. For low-light or backlit images, adaptive histogram equalization is used to improve local contrast. For color-shifted scenes, white balance and color constancy correction are performed. When compression artifacts are significant, wavelet or convolutional denoising is applied. For mildly blurred images, non-blind sharpening is used. For strongly blurred or motion stripe scenes, motion kernel estimation and non-blind deblurring are employed. For text-dense images, edge enhancement is added to improve readability for subsequent recognition. For portrait or document images, joint optimization of skin tone region noise suppression and detail preservation is performed. The enhancement process retains the original image and outputs the processed image and quality metrics, including sharpness score, contrast enhancement amount, and noise energy change, facilitating subsequent quality gating.

[0095] When integrating text data (after missing value imputation, outlier detection, and cleaning), speech data (after noise reduction), and image data (after enhancement) to generate data preprocessing results, standardized products are constructed based on the goals of reusability and auditability. For each composite record, a multimodal alignment index is established to record the temporal or semantic correspondence between each text segment, speech segment, and image frame, while also binding the quality indicators generated by cleaning and enhancement. The products consist of two parts: one is standardized data for downstream models, including uniformly coded text, audio with consistent sampling rate and standardized loudness, and images with standardized resolution and color correction; the other is process metadata, including anomaly and replacement lists, quality gating results, alignment relationships, and version numbers. To improve robustness and portability, the data preprocessing results use immutable object storage to store content and a retrieval index to store references. The index includes object identifiers, time windows, modal coverage, and quality thresholds. By marking and reprocessing counts, subsequent processes can automatically filter records based on quality thresholds and re-feed them as needed.

[0096] This embodiment utilizes partitioned archiving and a unified access protocol across text, voice, and image storage areas. This ensures consistent labeling and traceability of raw data from different channels before it enters the unified pipeline, reducing alignment discrepancies caused by source differences. Through time windows and object identifier aggregation of multimodal datasets, text, voice, and images form consistent composite records around the same object and event, providing stable input for subsequent representation construction. Missing value imputation, outlier detection, and cleaning operations systematically remove text noise, provide traceable imputation for field gaps, and standardize text scale and glyphs. Noise reduction and voice activity detection make voice segments more stable in terms of signal-to-noise ratio and temporal boundaries, reducing subsequent recognition and representation errors. Enhancement processing standardizes images in terms of exposure, color, and sharpness, significantly reducing drift in visual feature extraction. The final preprocessed data includes both standardized data and complete metadata, which can be directly used for downstream modeling and provide a basis for quality gating and auditing, thereby improving the reliability of cross-modal alignment and the repeatability of downstream feature extraction, and reducing error propagation in subsequent consistency verification and fusion stages.

[0097] In one embodiment, step S20 above includes:

[0098] S201, Obtain object historical structured information including object historical transaction data, object behavior log data, and object basic attribute data;

[0099] S202, Extract purchase frequency tags and transaction amount tags from the historical transaction data of the object;

[0100] S203, extract interaction activity tags, access frequency tags, and preference migration tags from the object behavior log data;

[0101] S204, Extract gender attribute labels and age attribute labels from the basic attribute data of the object;

[0102] S205, combine the purchase frequency tag, transaction amount tag, interaction activity tag, access frequency tag, preference migration tag, gender attribute tag and age attribute tag to generate a structured profile tag set.

[0103] In this embodiment, the step of acquiring the object's historical structured information, including historical transaction data, behavior log data, and basic attribute data, uses a three-element index of object unique identifier, time range, and source identifier, and loads the data according to a unified time zone and unified encoding. Historical transaction data covers the order master table, detail table, and payment records, with fields including at least order time, order status, order amount, currency, return / exchange flag, and payment completion flag. Behavior log data covers session start and end time, event type, event time, page or function identifier, dwell time, number of clicks, scroll depth, and device fingerprint. Basic attribute data covers gender, date of birth or age, document validity, region, and effective time. After loading, two-level alignment is performed: first, deduplication is performed at the object identifier dimension, and then window clipping is performed at the time dimension to ensure comparability of the three types of data within the same object and the same time window. To support subsequent metrics, an event classification dictionary and a transaction status mapping table are established, mapping event names and order statuses from multiple sources to finite sets respectively, avoiding caliber drift caused by synonyms.

[0104] When extracting purchase frequency tags from historical transaction data of an object, the effective counting unit is a transaction record where payment is completed but not fully cancelled, and the number of valid orders within the time window is counted. Multiple payment attempts for the same order are counted only once, and records of complete returns are counted as zero. To avoid interference from extreme duplicate orders, a collapsed window can be set at the object level to merge records with intervals less than a set threshold and similar products and amounts into a single purchase. Purchase frequency tags can be output as continuous values ​​and mapped to quantile buckets or rule buckets according to business criteria, such as low frequency, medium frequency, and high frequency. The mapping boundaries are stored in an entry table for audit traceability.

[0105] Transaction amount tags are based on standard currency amounts. First, multi-currency amounts are converted using the effective exchange rate. Then, abnormally large amounts are Windsor-processed or truncated using a threshold, resulting in three metrics: total amount, average transaction value, and median transaction value within the range. Total amount measures cumulative value, average transaction value reflects the intensity of a single transaction, and median transaction value suppresses the long-tail effect. Transaction amount tags retain continuous metrics and also output tiered tags, such as low, medium, and high amounts. The tier boundaries are consistent with the purchase frequency tags, configured using data-driven or rule-driven methods and incorporated into version control.

[0106] Before extracting interaction activity tags from object behavior log data, discrete events are aggregated into sessions. Adjacent events are grouped into the same session using inactivity timeout periods, and metrics such as session count, average session duration, and effective interaction event ratio are calculated. Interaction activity tags are represented as a weighted combination, with weights configured in metadata to balance the impact of frequency and duration. Continuous scores are output and synchronously mapped to level tags. Access frequency tags are based on the number of sessions or access days, normalized weekly or monthly to avoid incomparability due to differences in window length. Preference migration tags require constructing preference distribution vectors, representing the proportion of an object's contact with a category or function at different times as two vectors. The magnitude and direction of change between the vectors are calculated, for example, using cosine similarity or distribution difference measures, providing three states: upward shift, downward shift, and stable. The primary migration target category is recorded for interpretation.

[0107] When extracting gender and age attribute labels from basic object attribute data, the gender source is prioritized based on authoritative sources. In cases of multi-source conflicts, a strategy prioritizing credibility or the latest effective time is adopted, and a conflict list is maintained. Age is calculated from the birth date to the integer age at the end of the window; if only an age range is available, it falls directly into the corresponding interval. To ensure statistical stability, age attribute labels are output as interval labels, with interval divisions consistent with the business domain, such as minors, youth, middle-aged, and elderly. Interval boundaries are fixed and version-managed. Missing records or records with unresolved conflicts are uniformly output as unknown categories to prevent implicit missing data from infiltrating downstream processes.

[0108] When generating a structured profile tag set by combining purchase frequency tags, transaction amount tags, interaction activity tags, access frequency tags, preference migration tags, gender attribute tags, and age attribute tags, a consistency check is first performed to confirm that each tag is based on the same time window and the same object key. Then, scale and type are unified, and continuous measures and hierarchical tags are stored in parallel. Continuous measures are used for calculation scenarios, and hierarchical tags are used for retrieval and rule scenarios. To ensure traceability, each tag carries its source, calculation time, version number, and caliber number. When there are multiple sources for the same tag, a priority merging strategy is adopted. Priority indicates the source sorting and rollback rules. For example, if the transaction amount tag comes from both settlement and order, settlement takes precedence, and order takes precedence. Finally, the seven types of tags are written as an atomic entry into the structured profile tag set. The set follows a key-value pattern and a time-based activation pattern to ensure that tag queries and time-series analysis can be stably reproduced.

[0109] This embodiment achieves consistent expression of purchase frequency tags, transaction amount tags, interaction activity tags, access frequency tags, preference migration tags, gender attribute tags, and age attribute tags in terms of source, scope, and time dimensions by uniformly extracting and aligning historical transaction data, behavioral log data, and basic attribute data of objects. The tags retain continuous measurement while providing hierarchical levels, making it easy for different downstream tasks to reuse them directly. Conflict resolution and versioned recording ensure auditing and playback capabilities. The introduction of preference migration supplements the dynamic changes that static attributes cannot depict, thereby providing a stable, comparable, and traceable structured input for subsequent multimodal factor fusion and unified profile output, reducing ambiguity and data deviation in subsequent stages.

[0110] In one embodiment, step S30 above includes:

[0111] S301, create text prompts based on text characteristics, create voice prompts based on voice characteristics, and create image prompts based on image characteristics;

[0112] S302, combine the text prompts, voice prompts, and image prompts to generate a multimodal prompt set;

[0113] S303, extract text data from the data preprocessing result and convert the text data into a text token representation;

[0114] S304, extract speech data from the data preprocessing results and convert the speech data into a speech recognition token representation;

[0115] S305, extract image data from the data preprocessing results, and convert the image data into a visual token representation;

[0116] S306, Based on the text token representation, speech recognition token representation, and visual token representation and the multimodal cue word set, a unified representation sequence is constructed.

[0117] In this embodiment, when creating text prompts based on text characteristics, four elements are introduced: instruction tone, target object, extraction dimension, and output format. An example is setting a subject-verb structure for Chinese or multilingual templates based on dialogue corpora, limiting the slots to be focused on, such as sentiment, intent, product preference, and time cues, and binding field names and time windows with placeholders. To improve robustness, a template library and a collection of synonym rewrites are established, indexed by domain tags and language tags, and dual regular expression validation at the character and word levels is used to prevent placeholder omissions. Each text prompt is bound to a version number and effective time for easy auditing and gray-scale adjustments.

[0118] When creating voice prompts based on speech characteristics, acoustic dimensions such as pronunciation clarity, speech rate, pauses, stress, and background noise are transformed into controllable instruction phrases. Weights are used to indicate the relative importance of semantics and prosody. To cover dialect and accent differences, language type and accent tags are introduced, and error-tolerant expressions are set in the prompts, such as allowing synonyms or near-homophones, while specifying the need to retain emotional and role information. The output format of the prompts is uniformly a pairing of labels and timestamps, facilitating subsequent alignment with text or image timelines.

[0119] When creating image-based prompts, semantic instructions are organized around four dimensions: subject category, scene context, local attributes, and interactive elements. The hierarchical and fine-grained attributes of the objects to be identified are clearly defined, and bounding boxes or region mask placeholders are provided. For different sources such as screenshots, posters, product images, and interface flows, compression artifact tolerance and multi-objective decoupling strategies are given respectively. Geometric transformation invariance requirements are incorporated into the prompts to suppress descriptive drift caused by changes in viewpoint.

[0120] When generating a multimodal cue word set by combining text, voice, and image cue words, duplicate detection and conflict merging are performed first. Duplicate detection is completed using a dual-channel approach based on hashing and semantic similarity, while conflict merging follows the single responsibility principle, retaining only one effective cue word for the same extraction target. An index table is then constructed, with keys containing data source, language, domain, time window, and version, and values ​​containing the cue word text and constraint parameters. The set is distributed in read-only mode to ensure processing stability.

[0121] When extracting text data from the preprocessed data, uniform encoding and regularization are applied, preserving paragraph boundaries, dialogue turns, speaker markers, and timestamps. Text data is converted into text token representations using word-level segmentation, retaining out-of-vocabulary words, and using special markers to represent entities, amounts, and time expressions. To maintain comparability, maximum length and truncation strategies are set, and attention masks are generated to record valid positions, preserving intra-sentence and inter-sentence positional encodings. Inter-sentence encoding represents turn order, while intra-sentence encoding represents word order.

[0122] When extracting speech data from the preprocessing results, endpoint detection and energy normalization are performed first. Then, the automatic speech recognition engine is invoked to output transcribed text and word-level timestamps, while prosodic parameters such as fundamental frequency, energy, duration, and formants are extracted. When converting speech data into speech recognition token representations, sub-word-level encoding is performed on the transcribed text, and prosodic tokens are inserted at corresponding positions. For segments with uncertain recognition, confidence markers are inserted. Subsequences with confidence levels below a threshold are reserved as placeholders, and cross-segment attention is disabled on the mask to prevent noise propagation.

[0123] When extracting image data from the data preprocessing results, resolution and color space are unified according to the source, and the shooting or acquisition time is preserved and associated with a mapping key for the session or transaction. Two approaches are used to convert image data into a visual token representation. One approach is to divide the image into a fixed-scale grid and project it onto a vector space, obtaining a discrete index sequence through vector quantization, while attaching a two-dimensional positional code to each grid location. Alternatively, features can be extracted first using a visual encoder and then discretized using a productized vector quantization module to enhance reproducibility across batches. For images containing multiple targets, region indicator tokens are used to mark the boundaries of each target and the images are then stitched together in segments.

[0124] When constructing a unified representation sequence based on text token representation, speech recognition token representation, visual token representation, and a multimodal cue word set, a hierarchical concatenation strategy is adopted. Each modality is preceded by a modality start marker, followed by the insertion of a cue word fragment matching the current modality, and then the token sequence and positional encoding of that modality are concatenated. Modalities are connected by separator markers. To ensure temporal consistency, speech word-level timestamps and image acquisition times are mapped to a unified timeline, generating cross-modal alignment anchors. These anchors are inserted into the sequence with dedicated markers and carry normalized time values. To control sequence length, quotas and priorities are set, prioritizing the retention of high-weight fragments matching cue words. For excessively long segments, sliding window and overlapping sampling are used, and window markers are used to maintain reproducibility. The final output is a single-sequence representation containing a modality start, cue words, modality sequence, alignment anchors, and separator markers. It also outputs three types of auxiliary information: attention mask, modality segment identifier, and alignment index, to ensure consistent downstream processing and interpretable playback of the sequence.

[0125] This embodiment constructs prompt words for text, speech, and images respectively, forming a unified prompt word set. Combined with the cleaned and enhanced data preprocessing results, the three types of data are standardized into discrete tokens and aligned in a single time axis and a single sequence space. The unified representation sequence carries modal identity, temporal anchors, and prompt constraints in its structure. The model receiver does not need to perform caliber alignment and length pruning, and can directly perform joint modeling. The explicit occurrence of prompt conditions in the sequence improves the controllability and reproducibility of attention allocation. Noise and uncertainty are limited to a local range through confidence markers and attention masks, which reduces the probability of information loss and semantic drift during cross-modal fusion. This provides a more stable and auditable upstream input for subsequent fusion representation and derivative factor extraction.

[0126] In one embodiment, step S40 above includes:

[0127] S401, The unified representation sequence is input into the Transformer architecture of the multimodal model, and cross-modal joint modeling is performed in the Transformer architecture to obtain the fused representation;

[0128] S402, parse the feature information in the fused representation;

[0129] S403, based on the feature information, identify emotion-derived factors, intention-derived factors, product preference-derived factors, participation-derived factors, and behavior pattern-derived factors;

[0130] S404, combine the aforementioned emotion-derived factors, intention-derived factors, product preference-derived factors, participation-derived factors, and behavior pattern-derived factors to generate a candidate set of derivative factors.

[0131] In this embodiment, after the unified representation sequence enters the Transformer architecture of the multimodal model, it first completes the mapping from discrete labels to vector space through a shared embedding layer and modal segment identifiers, and adds it with the position vector and alignment anchor vector to form an input tensor. At the encoding end, layers of multi-head self-attention and feedforward networks are stacked. The self-attention accepts the unified representation sequence and attention mask. The mask prohibits attention propagation across invalid segments, ensuring that the temporal sequence and modal boundaries are not destroyed. In the cross-modal joint modeling process, two types of interaction units are introduced for the unified representation sequence. One is cross-modal gated attention, which uses modal segment identifiers to drive the gating coefficients to scale the attention scores of different modal channels. The other is the alignment anchor guided bias term, which maps the normalized time difference to an attention bias matrix to enhance the interaction intensity within the same event window. Layer normalization and residual connections run through each sub-layer to ensure the stability of deep propagation. The fusion representation is generated at the decoding end or in the aggregation head. It adopts a hybrid strategy of [CLS] aggregation, time window pooling and modal weighted averaging to output a two-level representation: the first level is a sequence-level fusion representation for global judgment, and the second level is a fragment-level fusion representation for local extraction and interpretation.

[0132] When parsing the feature information in the fusion representation, a multi-branch feature head is constructed to work in parallel. The sentiment branch takes the fragment-level fusion representation as input, uses bidirectional gating units to perform temporal smoothing on strong sentiment fragments, and then outputs sentiment polarity and intensity vectors through a sparse fully connected layer, while retaining confidence weights. The intent branch superimposes semantic trigger word query vectors on self-attention, assigns higher weights to fragments similar to task, need, rejection, and request trigger words, and finally outputs intent category and slot alignment information. The product preference branch performs prototype clustering on atomic vectors of word domains such as product name, category, function, and price, calculates the similarity between fragments and each prototype, and controls the degree of discretization with a temperature parameter. The engagement branch receives the joint encoding of dialogue turns, pauses, dwell time, and scrolling sequences in the fusion representation, captures behavioral density changes through multi-scale one-dimensional convolution, and outputs engagement scores and rhythm features. The behavior pattern branch uses sequence-to-sequence subgraph matching to map the fusion representation to primitive vectors of operations such as click, browse, consultation, and purchase, and mines patterns such as repetition, wandering, and rapid decision-making using differentiable path similarity measures. Each branch outputs a feature information tensor, which includes class distribution, intensity score, time boundary and confidence level.

[0133] When identifying sentiment-derived factors, intent-derived factors, product preference-derived factors, engagement-derived factors, and behavioral pattern-derived factors based on feature information, a unified candidate generation and denoising process is employed. Candidate generation begins with merging the time boundaries of branch outputs, using non-maximum suppression to remove overlapping intervals within the same category of candidates. Simultaneously, cross-branch consistency regularization is introduced to enhance candidate credibility when different branches generate mutually supporting evidence within the same time window. The denoising process provides a comprehensive score based on confidence level, length, contextual coherence, and alignment anchor coverage. Candidates with scores below a threshold are soft-masked and retained to prevent excessive discarding. Each type of derived factor includes a label name, intensity, time range, source modality weight, and explanatory fragment index for subsequent verification and fusion.

[0134] When generating a candidate set of derivative factors by combining emotion-derived factors, intention-derived factors, product preference-derived factors, engagement-derived factors, and behavior pattern-derived factors, the system first bins the candidates based on timelines and event identifiers, maintaining the same scenario or conversation scope within each bin. A hash mapping from category to candidate is established for each bin, and the source modality distribution, number of mutual verifications, and number of conflicts are recorded. For adjacent candidates of the same category, dynamic time warping and boundary re-evaluation are used to merge them into candidates with a longer span, avoiding fragmentation. For potential cross-category coupling, a lightweight causal graph is constructed to mark sequential and co-occurrence relationships, retaining only combinations whose causal strength or co-occurrence strength reaches a set threshold, thus completing the structured output of the candidate set of derivative factors. The set is accompanied by metadata, including generation time, model version, prompt word version, and threshold configuration, for traceability and reproduction.

[0135] This embodiment inputs a unified representation sequence into the Transformer architecture of a multimodal model and performs cross-modal joint modeling within it. It fuses representations within the same vector space, preserving semantic, prosodic, and visual cues from text, speech, and images. The parsed feature information is branched to cover five dimensions: sentiment, intent, product preference, engagement, and behavioral patterns. The candidate generation and denoising processes provide temporal boundaries, confidence levels, and cross-branch verification mechanisms. As a result, the derived factor candidate set significantly reduces noise propagation and spurious correlation risks, improves the pass rate and stability of subsequent consistency verification and weight calculation, and shortens the conversion path from the original multimodal input to elements usable for profile construction, making the overall processing chain more controllable and reproducible.

[0136] In one embodiment, step S50 above includes:

[0137] S501, extract text modality factors, speech modality factors, and image modality factors from the candidate set of derived factors;

[0138] S502, verify the consistency of the emotional features between the text modality factor and the speech modality factor;

[0139] S503, verify the consistency of the intent features between the text modality factor and the image modality factor;

[0140] S504, verify the consistency of product preference between the voice modality factor and the image modality factor;

[0141] S505, verify the consistency of the participation features between the text modality factor and the image modality factor;

[0142] S506, verify the consistency of the behavioral pattern features between the speech modality factor and the image modality factor;

[0143] S507: When the consistency of sentiment features meets the verification conditions, sentiment-derived factors are retained;

[0144] S508, when the consistency of intent features meets the verification conditions, the intent derivation factor is retained;

[0145] S509, when the product tendency consistency meets the verification conditions, the product tendency derived factor is retained;

[0146] S510: When the consistency of participation features meets the verification conditions, the participation-derived factors are retained.

[0147] S511, When the consistency of behavioral pattern features meets the verification conditions, retain the behavioral pattern derived factors;

[0148] S512 combines emotional derivative factors, intention derivative factors, product preference derivative factors, participation derivative factors, and behavioral pattern derivative factors to generate a set of consistent factors.

[0149] In this embodiment, after the candidate set of derived factors enters the consistency verification stage, the sorting and indexing of text modality factors, speech modality factors, and image modality factors are first completed. Sorting uses the source modality and time boundary recorded in the candidates as keys to construct an inverted index mapping from modality to candidate, and generates three types of vector representations and structured attributes for each candidate. For text modality factors, text semantic vectors, sentiment intensity, intent slots, product word prototype similarity, and engagement statistics are extracted. For speech modality factors, acoustic embeddings, prosodic features, emotional acoustic indicators, and alignment positions of semantic recognition segments are extracted. For image modality factors, visual embeddings, product prototype similarity distribution, regional attention weights, and duration and scrolling amplitude related to image interactions in the behavioral sequence are extracted. All candidates retain timestamps, source session identifiers, confidence levels, and mutual verification counts to support subsequent cross-modal pairing and statistics.

[0150] Sentiment feature consistency verification uses text modality factors and speech modality factors as paired objects. First, window alignment is performed on a unified time axis. Dynamic time warping is used to align the boundaries of speech frame sequences and text segments to a common window. Then, two types of indicators are calculated. The first type is semantic and acoustic sentiment consistency, measured using a cosine similarity using binary vectors of sentiment polarity and intensity, supplemented by a two-dimensional spatial distance between valence and arousal. The second type is temporal consistency, detecting whether the peak sentiment occurrence windows overlap and calculating the overlap rate. The comprehensive score is obtained through a weighted geometric mean, with weights determined by the calibration curve based on the validation set during the training phase. The verification conditions consist of three constraints: the comprehensive score is not lower than a threshold, the overlap rate is not lower than a coverage threshold, and the single-modality confidence is not lower than a minimum confidence threshold. When the verification conditions are met, the corresponding sentiment-derived factor is marked as retained, and the score and alignment window at the time of success are recorded.

[0151] Intent feature consistency verification uses text modality factors and image modality factors as paired objects. First, intent slots in the text are mapped to the business intent ontology, and target categories and attributes in the image are mapped to the same ontology node or its sibling nodes. Then, ontology path distance and label similarity are calculated. Simultaneously, the consistency of the behavioral context is evaluated, detecting whether the intent trigger word and the region of interest in the image are in the same session round or the same operation path branch. The comprehensive score consists of three parts: semantic ontology similarity, label similarity, and contextual consistency. Threshold gating and temperature scaling are used for calibration. When the verification conditions are met, the intent-derived factors are retained and the matched ontology node number is written.

[0152] Product preference consistency verification uses voice modality factors and image modality factors as paired objects. The similarity distribution of product prototypes in speech-to-text and the distribution of product prototypes obtained from image retrieval are tested for distribution consistency, using a joint measure of Earth's distance traveled and log-likelihood ratio. Simultaneously, the consistency coverage ratio of price range, functional attributes, and brand attributes is checked. If the distribution measure passes the threshold and the attribute coverage ratio meets the standard, the product preference is determined to be consistent, and the product preference derived factors are retained along with the most likely product prototype number and attribute set. If it fails, it is kept in a pending review status for subsequent joint verification with text modality factors.

[0153] Engagement feature consistency verification uses text modality factors and image modality factors as paired objects. A behavioral density time series is constructed: for text, the data comes from the number of rounds, question depth, and proportion of long sentences; for images, it comes from dwell time, scrolling speed, and number of interactions. Multi-scale correlation coefficients are used to evaluate the consistency of the two density curves, and the proportion of intersection time in high-density segments is calculated. When the overall score meets the verification conditions, engagement-derived factors are retained, and rhythm type labels are recorded, such as continuous high density or phased high density, to support subsequent profile segmentation.

[0154] Behavioral pattern feature consistency verification uses speech modal factors and image modal factors as paired objects. First, the operation intent sequence on the speech side and the page jump sequence on the image side are mapped to operation primitives. Then, differentiable path matching is used to calculate the edit distance and the length of the longest common subsequence between the two primitive sequences, and a window sliding method is used to find the co-occurrence of repeating patterns or wandering patterns. If the path consistency index and co-occurrence frequency meet the verification conditions, the behavioral pattern derived factors are retained and the pattern type and key inflection point positions are output.

[0155] After the retention rules are executed, factors derived from emotion, intention, product preference, participation, and behavior patterns are aggregated to form a consistent factor set. During the aggregation process, factors with the same name are merged over time and conflict resolved. Merging uses boundary reassessment and intensity-weighted averaging, while conflict resolution uses a cascaded criterion of confidence priority and mutual verification count priority. For cross-class strongly coupled factors, causal orientation or co-occurrence relationships are recorded to avoid double counting in subsequent weight allocation. The consistent factor set, along with pass records, failure records, and threshold configurations, is output to ensure traceability and verifiability.

[0156] This embodiment implements targeted consistency verification of three pairs of two modalities within a unified time axis and unified ontology space. Semantic, acoustic, and visual evidence are aligned at the window level and judged using multi-indicator joint gating, effectively filtering out pseudo-factors and conflicting factors introduced by single-modal noise. After passing the verification conditions, the five factors—emotion, intent, product preference, engagement, and behavioral patterns—maintain a structured expression of time boundaries and source weights, providing stable input for subsequent fusion and weight calculation, reducing error propagation and duplicate counting, improving the accuracy and interpretability of profile construction, and increasing the downstream fusion convergence speed and consistency output ratio under the same data scale.

[0157] In one embodiment, step S60 above includes:

[0158] S601, extract emotional derivative factors, intention derivative factors, product preference derivative factors, participation derivative factors and behavioral pattern derivative factors from the set of consistent factors;

[0159] S602, extract purchase frequency tags, transaction amount tags, interaction activity tags, access frequency tags, preference migration tags, gender attribute tags, and age attribute tags from the structured profile tag set;

[0160] S603, count the number of emotion tags for emotion-derived factors, the number of intention tags for intention-derived factors, the number of product preference tags for product preference-derived factors, the number of participation tags for participation-derived factors, and the number of behavior pattern tags for behavior pattern-derived factors;

[0161] S604, based on the number of emotion markers, the number of intention markers, the number of product preference markers, the number of engagement markers, and the number of behavior pattern markers, determine the emotion weight value, the intention weight value, the product preference weight value, the engagement weight value, and the behavior pattern weight value;

[0162] S605, apply the sentiment weight value to the sentiment derivative factor to generate a weighted sentiment factor;

[0163] S606, Apply the intent weight value to the intent derived factor to generate a weighted intent factor;

[0164] S607, apply the product preference weight value to the product preference derived factor to generate a weighted product preference factor;

[0165] S608, apply the participation weight value to the participation derived factor to generate a weighted participation factor;

[0166] S609, Apply the behavioral pattern weight values ​​to the behavioral pattern derived factors to generate weighted behavioral pattern factors;

[0167] S610, integrates the weighted sentiment factor, weighted intention factor, weighted product preference factor, weighted participation factor, weighted behavior pattern factor, purchase frequency tag, transaction amount tag, interaction activity tag, access frequency tag, preference migration tag, gender attribute tag and age attribute tag to generate a unified profile system;

[0168] S611, Extract the object profile tag set from the unified profile system;

[0169] S612, output the set of object profile tags.

[0170] In this embodiment, after the consistent factor set and the structured profile tag set enter the fusion stage, the isomorphic representation and indexing of the two types of data are first completed. The consistent factor set includes sentiment-derived factors, intent-derived factors, product preference-derived factors, participation-derived factors, and behavior pattern-derived factors. Each factor carries a time boundary, source modality, confidence level, and mutual verification count. The structured profile tag set includes purchase frequency tags, transaction amount tags, interaction activity tags, access frequency tags, preference migration tags, gender attribute tags, and age attribute tags. To eliminate differences in heterogeneous fields, a unified tag ontology and feature dictionary are first established. The five types of derived factors are mapped to higher-level classes such as behavior, preference, and emotion, and the seven types of structured tags are mapped to higher-level classes such as demographic attributes and transaction behavior. A unified key value is generated for the entries on both sides, including tag encoding, value range, statistical window, and confidence interval, to ensure that subsequent calculations can be performed directly in the vector space.

[0171] When extracting five types of derived factors from the consistent factor set, deduplication is performed according to the tag ontology encoding, and the highest confidence entries are retained. Records with adjacent times and the same source are merged, using boundary reestimation and intensity-weighted averaging as merging rules to reduce duplicate counting. When extracting seven types of tags from the structured profile tag set, the time window is unified to the most recent complete observation period, and differences in definitions across different business systems are aligned using a definition mapping matrix. For example, differences in the definition of purchase frequency across different systems are addressed through threshold conversion and distribution recalibration to achieve consistency. After extraction and definition alignment, the tag count is then performed, counting the number of sentiment tags, intent tags, product preference tags, engagement tags, and behavioral pattern tags for each of the five derived factors. The statistical objects are derived from records that have passed cross-modal consistency verification. Counting is performed by aggregating by time window and session identifier; repeated triggers within the same window are counted only once to prevent inflation caused by channel jitter. For continuous activation across windows, a continuous segment length reduction coefficient is used to suppress the bias of long-tail accumulation on the weights.

[0172] Weights are determined based on the normalized number of labelings. L1 normalization is performed on the count vectors of each user across the five derived factors to obtain sentiment weights, intent weights, product preference weights, engagement weights, and behavioral pattern weights. To improve stability, Dirichlet prior smoothing is introduced to avoid extreme weights for users with low counts. A time decay function is also introduced, with a higher proportion of recent labels to reflect the time-varying nature of interests and emotions. If there are users with sparse data or from a single source, group statistical backtracking is triggered, linearly mixing individual weights with the prior weights of the same group using a confidence coefficient determined by the user's effective observation duration and mutual verification counts.

[0173] In the weighting application phase, sentiment weights are applied to sentiment-derived factors, and intention weights are applied to intention-derived factors, thereby generating weighted sentiment factors, weighted intention factors, weighted product preference factors, weighted participation factors, and weighted behavioral pattern factors. The weighting method employs a two-level mapping: the first level scales the amplitude along the factor intensity dimension, making the intensity proportional to the weight; the second level weights the edge weights in the fusion graph model, ensuring that the connections between factors and structured labels have an influence consistent with their weights. To prevent excessively high weights for any one factor leading to single-class dominance, a temperature parameter is introduced for softening, ensuring that the five types of factors maintain a comparable scale during the fusion phase.

[0174] The integration phase constructs a heterogeneous graph representation of a unified user profile system. The graph uses the user as the central node, structured tags and weighted derived factors as attribute nodes, and behavior, preferences, emotions, demographic attributes, and transaction behavior as relationship types. Weighted sentiment factors, weighted intent factors, weighted product preference factors, weighted participation factors, weighted behavioral pattern factors, and purchase frequency tags, transaction amount tags, interaction activity tags, access frequency tags, preference migration tags, gender attribute tags, and age attribute tags are connected to the central node. Edge weights consist of three parts: weighting strength, data freshness, and source credibility. Freshness is given by a time decay function, and source credibility comes from cross-modal verification pass rate and data source quality assessment score. Conflict resolution is performed on multiple edges under the same semantic meaning. If there are opposing tendencies or numerical contradictions, edges are selected and retained according to the cascaded criteria of edge weight, mutual verification count, and data freshness. Unretained edges enter the candidate pool with low weight for subsequent incremental update verification.

[0175] After the unified profile system is generated, a set of object profile tags is output based on the task perspective. The output is accomplished using a perspective-driven selector: for a preference perspective, it prioritizes outputting product-related nodes and their strongly coupled purchase frequency and transaction amount; for an interaction perspective, it prioritizes outputting participation and interaction activity levels and their time patterns; and for an insight perspective, it outputs a joint summary of sentiment and intent, with explanatory variables stratified according to age and gender attributes. The selector follows the shortest explanatory path principle on the graph, prioritizing derived factor nodes that can be connected to key structured tags with the shortest path, and outputs the source, time range, edge weights, and confidence intervals of each tag to support auditing and verification. The final result forms a structured list of object profile tags, which can be directly consumed by downstream profile browsing, recommendation, or marketing decisions.

[0176] Example Explanation: In the healthcare field, multimodal profiles can be constructed based on individual health records and lifestyle data to support chronic disease management, health interventions, and personalized health and wellness services. First, a multimodal dataset is formed by collecting and integrating data from multiple sources, including raw information generated by individuals from wearable devices, home health monitoring instruments, health consultation voice recordings, and health-related images. This data is stored separately in text, voice, and image storage areas. After extraction, it is integrated into a single dataset and undergoes missing value imputation, outlier detection and cleaning, voice denoising, and image enhancement to obtain data preprocessing results that support subsequent modeling.

[0177] Building upon this foundation, a set of structured profile tags is extracted by combining historical structured health information. This information includes an individual's past health check data, lifestyle logs, and basic attribute records, such as daily exercise frequency, sleep duration, and nutritional intake. By parsing this information, purchase frequency tags correspond to health product purchasing habits, transaction amount tags correspond to nutritional consumption levels, interaction activity tags and access frequency tags reflect the user's interaction behavior with the health management platform, preference migration tags record lifestyle change trends, and gender and age attribute tags provide basic demographic characteristics. The combination of these tags forms a complete set of structured profile tags.

[0178] Next, a multimodal cue word set is introduced, mapping text cue words to diet and exercise-related text, voice cue words to voice consultation content, and image cue words to health images or exercise posture pictures. By mapping these cue words to preprocessed data, text tokens, voice recognition tokens, and visual tokens are generated respectively, and then fused into a unified representation sequence. This unified representation sequence is input into the Transformer architecture of the multimodal large-scale model to perform cross-modal joint modeling and generate a fused representation. Subsequently, the feature information in the fused representation is analyzed to identify emotion-derived factors reflecting healthy emotional states, intention-derived factors representing health behavior intentions, product preference-derived factors representing preferences for health services or equipment, participation-derived factors corresponding to the degree of participation in health activities, and behavior pattern-derived factors characterizing the regularity of diet, exercise, and rest, and these are combined into a candidate set of derivative factors.

[0179] For this candidate set, further cross-modal consistency verification is performed. The authenticity of health emotions is assessed by comparing the consistency between text modal factors and speech modal factors; the rationality of intention features is assessed by comparing text and image modal factors; the consistency of health service selection is verified by comparing speech and image modal factors; and the degree of conformity between engagement and behavioral patterns across different modalities is also checked. Only factors that meet the consistency conditions are retained, and finally, a consistent factor set is generated.

[0180] In the fusion of the consistent factor set and the structured profile tag set, the weights of different derived factors are determined by statistically analyzing the number of times they are labeled. For example, frequently exhibited health behavior patterns will receive higher weight values. These weights are then applied to the five categories of derived factors to generate weighted factors. Subsequently, these weighted factors are integrated with structured profile tags such as health consumption, interaction frequency, preference migration, and demographic attributes to construct a unified profile system. The final output set of object profile tags reflects the user's overall health status, such as the correlation between nutritional consumption trends and exercise participation, and the coupling relationship between health emotions and lifestyle habits, thus providing a solid foundation for health intervention, health and wellness service recommendations, and long-term health tracking.

[0181] In the fintech business, this approach can address the challenges of refined customer management and service recommendations. In practical applications, the first step is to collect and integrate data from multiple sources. This data includes traditional structured information such as customer transaction records, fund transfer records, and payment habit logs, as well as unstructured information such as customer expressions during customer service voice interactions, click behavior when browsing products on mobile apps, and even uploaded contract or invoice image files. All this data is integrated into a multimodal dataset, and after missing value imputation, outlier cleaning, voice signal denoising, and image enhancement, a reliable data preprocessing result is formed that can be used for modeling.

[0182] Building upon this foundation, a set of structured profile tags is extracted by combining the customer's historical structured information. For example, transaction flow data can be transformed into purchase frequency tags and transaction amount tags, behavioral log data can be transformed into interaction activity tags, access frequency tags, and preference migration tags, while the customer's basic attribute data provides gender and age attribute tags. These tags collectively construct a basic profile reflecting the customer's consumption patterns and interactive behaviors in financial services.

[0183] To further enhance the expressive power of customer profiles, the system generates a multimodal cue word set, mapping text cue words to expressions of financial needs, voice cue words to customer service interaction records, and image cue words to images of invoices or contracts. These are then mapped to preprocessed data to form a unified representation sequence. This representation sequence is input into the Transformer architecture of the multimodal large-scale model, performing cross-modal joint modeling to obtain a fused representation. After parsing, the fused representation identifies emotion-derived factors, intent-derived factors, product preference-derived factors, engagement-derived factors, and behavioral pattern-derived factors. These factors characterize customers' emotional attitudes towards financial products, potential intentions, risk preferences, interaction levels, and fund management behaviors, and are combined into a candidate set of derived factors.

[0184] Within the candidate set of derivative factors, cross-modal consistency verification is required. For example, comparing customer emotional expressions in text and voice modalities helps determine their true level of investment confidence; comparing text and image modalities verifies the consistency of customer intent when filling out application materials and making verbal statements; comparing voice and image modalities verifies whether the customer's verbally expressed risk preferences match the type of financial product selected in the contract; and verifying the cross-modal consistency of participation and behavioral patterns identifies whether customers consistently maintain their trading frequency or rhythm of financial activities. Factors that meet the consistency criteria are retained, forming a consistent factor set.

[0185] In the subsequent integration phase, the consistent factor set is combined with the structured profile tag set. At this point, by statistically analyzing the number of times various derived factors are labeled and calculating their weights, weighted factors that better reflect customer behavior patterns are obtained. Weighted sentiment factors reflect customers' emotional tendencies in financial activities; weighted intention factors reflect customers' potential demand for future products; weighted product preference factors reflect the intensity of interest in different financial products; weighted engagement factors characterize the depth and stickiness of interactions; and weighted behavioral pattern factors express long-term capital operation habits. These factors are integrated with the structured tags to generate a unified customer profile system, ultimately outputting a set of object profile tags.

[0186] This holistic process addresses the shortcomings of traditional customer profiling in the fintech field, which relies too heavily on transaction amounts and basic attributes and lacks the ability to capture customer emotions, intentions, and dynamic behavioral patterns. By integrating multimodal data and performing cross-modal consistency checks, customer profiles become more comprehensive and possess greater interpretability and reliability in applications such as financial service recommendations, credit assessment, risk monitoring, and precision marketing.

[0187] This embodiment unifies the representation of five types of derived factors and seven types of structured labels after cross-modal verification within the same ontology and graph space. It uses normalized weight determination driven by label counts, along with mechanisms such as time decay, prior smoothing, and group backoff to constrain weight stability. This approach suppresses single-source noise and count inflation during the fusion phase, eliminates biases caused by differences in caliber, and achieves a traceable, interpretable, and updatable unified profiling system. After the weighted factors and structured labels are fused according to edge weights, the output object profile label set maintains consistent scale and time alignment across the preference, behavior, and emotion dimensions, reducing conflicts and duplicate counts. This provides high-confidence input for downstream identification and decision-making, and improves convergence speed and label stability under the same data scale.

[0188] In one embodiment, an object image tag set generation apparatus is provided, which corresponds one-to-one with the object image tag set generation method in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the image tag set generation device of the present invention. The modules include a data acquisition and preprocessing module 10, a tag extraction module 20, a prompt word construction module 30, a factor extraction module 40, a consistency verification module 50, and an image fusion output module 60. Detailed descriptions of each functional module are as follows:

[0189] The data acquisition and preprocessing module 10 is used to acquire and integrate multi-source data, generate a multimodal data set, perform cleaning and enhancement operations on the multimodal data set, and generate data preprocessing results.

[0190] Tag extraction module 20 is used to extract a set of structured profile tags based on the object's historical structured information;

[0191] The prompt word construction module 30 is used to obtain a multimodal prompt word set and construct a unified representation sequence based on the data preprocessing result and the multimodal prompt word set;

[0192] The factor extraction module 40 is used to input the unified representation sequence into the multimodal model to obtain a fusion representation, and extract a candidate set of derived factors from the fusion representation;

[0193] The consistency verification module 50 is used to perform cross-modal consistency verification on the cross-modal data in the candidate set of derived factors to obtain a consistent factor set;

[0194] The portrait fusion output module 60 is used to fuse the set of consistency factors with the set of structured portrait tags to generate a unified portrait system and output the set of object portrait tags in the unified portrait system.

[0195] In one embodiment, the data acquisition and preprocessing module 10 is specifically used for:

[0196] Collect text data, voice data, and image data; store the text data in a text storage area, store the voice data in a voice storage area, and store the image data in an image storage area;

[0197] Data is extracted from the text storage area, voice storage area and image storage area, and the extracted data is integrated to generate a multimodal data set;

[0198] Perform missing value imputation, outlier detection, and cleaning operations on the text data in the multimodal dataset;

[0199] Noise reduction processing is performed on the speech data in the multimodal dataset;

[0200] Enhancement processing is performed on the image data in the multimodal dataset;

[0201] The data preprocessing results are generated by integrating text data after missing value imputation, outlier detection and cleaning, speech data after noise reduction, and image data after enhancement.

[0202] In one embodiment, the tag extraction module 20 is specifically used for:

[0203] Obtain historical structured information about an object, including historical transaction data, behavior log data, and basic attribute data.

[0204] Extract purchase frequency tags and transaction amount tags from the historical transaction data of the object;

[0205] Extract interaction activity tags, access frequency tags, and preference migration tags from the object behavior log data;

[0206] Extract gender and age attribute labels from the object's basic attribute data;

[0207] The purchase frequency tag, transaction amount tag, interaction activity tag, access frequency tag, preference migration tag, gender attribute tag, and age attribute tag are combined to generate a structured profile tag set.

[0208] In one embodiment, the prompt word construction module 30 is specifically used for:

[0209] Text prompts are created based on text characteristics; voice prompts are created based on voice characteristics; and image prompts are created based on image characteristics.

[0210] The text prompts, voice prompts, and image prompts are combined to generate a multimodal prompt set;

[0211] Extract text data from the data preprocessing results and convert the text data into a text token representation;

[0212] Extract speech data from the data preprocessing results and convert the speech data into a speech recognition token representation;

[0213] Extract image data from the data preprocessing results and convert the image data into a visual token representation;

[0214] Based on the text token representation, speech recognition token representation, and visual token representation, and the multimodal cue word set, a unified representation sequence is constructed.

[0215] In one embodiment, the factor extraction module 40 is specifically used for:

[0216] The unified representation sequence is input into the Transformer architecture of the multimodal model, and cross-modal joint modeling is performed in the Transformer architecture to obtain the fused representation;

[0217] Analyze the feature information in the fused representation;

[0218] Based on the aforementioned feature information, we can identify emotion-derived factors, intention-derived factors, product preference-derived factors, engagement-derived factors, and behavioral pattern-derived factors.

[0219] The aforementioned emotion-derived factors, intention-derived factors, product preference-derived factors, participation-derived factors, and behavior pattern-derived factors are combined to generate a candidate set of derivative factors.

[0220] In one embodiment, the consistency verification module 50 is specifically used for:

[0221] Extract text modality factors, speech modality factors, and image modality factors from the candidate set of derived factors;

[0222] Verify the consistency of the emotional features between the text modality factor and the speech modality factor;

[0223] Verify the consistency of the intent features between the text modality factor and the image modality factor;

[0224] Verify the consistency of product preference between the voice modality factor and the image modality factor;

[0225] Verify the consistency of the participation features between the text modality factor and the image modality factor;

[0226] Verify the consistency of behavioral pattern features between the speech modality factor and the image modality factor;

[0227] When the consistency of sentiment features meets the verification conditions, the sentiment-derived factors are retained;

[0228] When the consistency of intent features meets the verification conditions, the intent-derived factors are retained;

[0229] When the product tendency consistency meets the verification conditions, the product tendency derived factors are retained;

[0230] When the consistency of participation features meets the verification conditions, the participation-derived factors are retained;

[0231] When the consistency of behavioral pattern features meets the verification conditions, the behavioral pattern derived factors are retained.

[0232] Combine emotional derivative factors, intention derivative factors, product preference derivative factors, participation derivative factors, and behavioral pattern derivative factors to generate a set of consistent factors.

[0233] In one embodiment, the image fusion output module 60 is specifically used for:

[0234] Extract emotion-derived factors, intention-derived factors, product preference-derived factors, engagement-derived factors, and behavior pattern-derived factors from the set of consistent factors;

[0235] Extract purchase frequency tags, transaction amount tags, interaction activity tags, access frequency tags, preference migration tags, gender attribute tags, and age attribute tags from the structured profile tag set;

[0236] The number of emotion markers for emotion-derived factors, the number of intention markers for intention-derived factors, the number of product preference markers for product preference-derived factors, the number of participation markers for participation-derived factors, and the number of behavior pattern markers for behavior pattern-derived factors were counted.

[0237] Based on the number of emotion markers, intent markers, product preference markers, engagement markers, and behavioral pattern markers, determine the emotion weight value, intent weight value, product preference weight value, engagement weight value, and behavioral pattern weight value;

[0238] Apply the sentiment weight values ​​to the sentiment-derived factors to generate a weighted sentiment factor;

[0239] Apply the intent weight value to the intent-derived factor to generate a weighted intent factor;

[0240] Apply the product preference weight values ​​to the product preference derived factors to generate a weighted product preference factor;

[0241] Apply the participation weight values ​​to the participation-derived factors to generate a weighted participation factor;

[0242] Apply the behavioral pattern weight values ​​to the behavioral pattern derived factors to generate weighted behavioral pattern factors;

[0243] By integrating the weighted sentiment factors, weighted intention factors, weighted product preference factors, weighted participation factors, weighted behavior pattern factors, purchase frequency tags, transaction amount tags, interaction activity tags, access frequency tags, preference migration tags, gender attribute tags, and age attribute tags, a unified profile system is generated.

[0244] Extract the set of object profile tags from the unified profile system;

[0245] Output the set of object profile tags.

[0246] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a server-side method for generating object profile tag sets.

[0247] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of an object image tag set generation method.

[0248] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0249] Collect and integrate multi-source data to generate a multimodal dataset, perform cleaning and enhancement operations on the multimodal dataset, and generate data preprocessing results;

[0250] Extracting a set of structured profile tags based on object historical structured information;

[0251] Obtain a set of multimodal cue words, and construct a unified representation sequence based on the data preprocessing results and the set of multimodal cue words;

[0252] The unified representation sequence is input into a multimodal model to obtain a fusion representation, and a candidate set of derived factors is extracted from the fusion representation;

[0253] Perform cross-modal consistency verification on the cross-modal data in the candidate set of derived factors to obtain a consistent factor set;

[0254] The set of consistent factors is merged with the set of structured profile tags to generate a unified profile system, and the set of object profile tags in the unified profile system is output.

[0255] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0256] Collect and integrate multi-source data to generate a multimodal dataset, perform cleaning and enhancement operations on the multimodal dataset, and generate data preprocessing results;

[0257] Extracting a set of structured profile tags based on object historical structured information;

[0258] Obtain a set of multimodal cue words, and construct a unified representation sequence based on the data preprocessing results and the set of multimodal cue words;

[0259] The unified representation sequence is input into a multimodal model to obtain a fusion representation, and a candidate set of derived factors is extracted from the fusion representation;

[0260] Perform cross-modal consistency verification on the cross-modal data in the candidate set of derived factors to obtain a consistent factor set;

[0261] The set of consistent factors is merged with the set of structured profile tags to generate a unified profile system, and the set of object profile tags in the unified profile system is output.

[0262] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0263] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0264] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0265] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for generating an object profile tag set, characterized in that, Includes the following steps: Collect and integrate multi-source data to generate a multimodal dataset, perform cleaning and enhancement operations on the multimodal dataset, and generate data preprocessing results; Extracting a set of structured profile tags based on object historical structured information; Obtain a set of multimodal cue words, and construct a unified representation sequence based on the data preprocessing results and the set of multimodal cue words; The unified representation sequence is input into a multimodal model to obtain a fusion representation, and a candidate set of derived factors is extracted from the fusion representation; Perform cross-modal consistency verification on the cross-modal data in the candidate set of derived factors to obtain a consistent factor set; The set of consistent factors is merged with the set of structured profile tags to generate a unified profile system, and the set of object profile tags in the unified profile system is output.

2. The method for generating an object profile tag set as described in claim 1, characterized in that, Collect and integrate multi-source data to generate a multimodal dataset. Perform cleaning and enhancement operations on the multimodal dataset to generate data preprocessing results, including: Collect text data, voice data, and image data; store the text data in a text storage area, store the voice data in a voice storage area, and store the image data in an image storage area; Data is extracted from the text storage area, voice storage area and image storage area, and the extracted data is integrated to generate a multimodal data set; Perform missing value imputation, outlier detection, and cleaning operations on the text data in the multimodal dataset; Noise reduction processing is performed on the speech data in the multimodal dataset; Enhancement processing is performed on the image data in the multimodal dataset; The data preprocessing results are generated by integrating text data after missing value imputation, outlier detection and cleaning, speech data after noise reduction, and image data after enhancement.

3. The method for generating an object profile tag set as described in claim 1, characterized in that, Based on the object's historical structured information, a set of structured profile tags is extracted, including: Obtain historical structured information about an object, including historical transaction data, behavior log data, and basic attribute data. Extract purchase frequency tags and transaction amount tags from the historical transaction data of the object; Extract interaction activity tags, access frequency tags, and preference migration tags from the object behavior log data; Extract gender and age attribute labels from the object's basic attribute data; The purchase frequency tag, transaction amount tag, interaction activity tag, access frequency tag, preference migration tag, gender attribute tag, and age attribute tag are combined to generate a structured profile tag set.

4. The method for generating an object profile tag set as described in claim 1, characterized in that, Obtain a multimodal cue word set, and based on the data preprocessing results and the multimodal cue word set, construct a unified representation sequence, including: Text prompts are created based on text characteristics; voice prompts are created based on voice characteristics; and image prompts are created based on image characteristics. The text prompts, voice prompts, and image prompts are combined to generate a multimodal prompt set; Extract text data from the data preprocessing results and convert the text data into a text token representation; Extract speech data from the data preprocessing results and convert the speech data into a speech recognition token representation; Extract image data from the data preprocessing results and convert the image data into a visual token representation; Based on the text token representation, speech recognition token representation, and visual token representation, and the multimodal cue word set, a unified representation sequence is constructed.

5. The method for generating an object profile tag set as described in claim 1, characterized in that, The unified representation sequence is input into a multimodal model to obtain a fused representation, and a candidate set of derived factors is extracted from the fused representation, including: The unified representation sequence is input into the Transformer architecture of the multimodal model, and cross-modal joint modeling is performed in the Transformer architecture to obtain the fused representation; Analyze the feature information in the fused representation; Based on the aforementioned feature information, we can identify emotion-derived factors, intention-derived factors, product preference-derived factors, engagement-derived factors, and behavioral pattern-derived factors. The aforementioned emotion-derived factors, intention-derived factors, product preference-derived factors, participation-derived factors, and behavior pattern-derived factors are combined to generate a candidate set of derivative factors.

6. The method for generating an object profile tag set as described in claim 1, characterized in that, Perform cross-modal consistency verification on the cross-modal data in the candidate set of derived factors to obtain a consistent factor set, including: Extract text modality factors, speech modality factors, and image modality factors from the candidate set of derived factors; Verify the consistency of the emotional features between the text modality factor and the speech modality factor; Verify the consistency of the intent features between the text modality factor and the image modality factor; Verify the consistency of product preference between the voice modality factor and the image modality factor; Verify the consistency of the participation features between the text modality factor and the image modality factor; Verify the consistency of behavioral pattern features between the speech modality factor and the image modality factor; When the consistency of sentiment features meets the verification conditions, the sentiment-derived factors are retained; When the consistency of intent features meets the verification conditions, the intent-derived factors are retained; When the product tendency consistency meets the verification conditions, the product tendency derived factors are retained; When the consistency of participation features meets the verification conditions, the participation-derived factors are retained; When the consistency of behavioral pattern features meets the verification conditions, the behavioral pattern derived factors are retained. Combine emotional derivative factors, intention derivative factors, product preference derivative factors, participation derivative factors, and behavioral pattern derivative factors to generate a set of consistent factors.

7. The method for generating an object profile tag set as described in claim 1, characterized in that, The set of consistency factors is merged with the set of structured profile tags to generate a unified profile system, and the set of object profile tags in the unified profile system is output, including: Extract emotion-derived factors, intention-derived factors, product preference-derived factors, engagement-derived factors, and behavior pattern-derived factors from the set of consistent factors; Extract purchase frequency tags, transaction amount tags, interaction activity tags, access frequency tags, preference migration tags, gender attribute tags, and age attribute tags from the structured profile tag set; The number of emotion markers for emotion-derived factors, the number of intention markers for intention-derived factors, the number of product preference markers for product preference-derived factors, the number of participation markers for participation-derived factors, and the number of behavior pattern markers for behavior pattern-derived factors were counted. Based on the number of emotion markers, intent markers, product preference markers, engagement markers, and behavioral pattern markers, determine the emotion weight value, intent weight value, product preference weight value, engagement weight value, and behavioral pattern weight value; Apply the sentiment weight values ​​to the sentiment-derived factors to generate a weighted sentiment factor; Apply the intent weight value to the intent-derived factor to generate a weighted intent factor; Apply the product preference weight values ​​to the product preference derived factors to generate a weighted product preference factor; Apply the participation weight values ​​to the participation-derived factors to generate a weighted participation factor; Apply the behavioral pattern weight values ​​to the behavioral pattern derived factors to generate weighted behavioral pattern factors; By integrating the weighted sentiment factors, weighted intention factors, weighted product preference factors, weighted participation factors, weighted behavior pattern factors, purchase frequency tags, transaction amount tags, interaction activity tags, access frequency tags, preference migration tags, gender attribute tags, and age attribute tags, a unified profile system is generated. Extract the set of object profile tags from the unified profile system; Output the set of object profile tags.

8. An object profile tag set generation device, characterized in that, The object profile tag set generation device includes: The data acquisition and preprocessing module is used to acquire and integrate multi-source data, generate a multimodal data set, perform cleaning and enhancement operations on the multimodal data set, and generate data preprocessing results. The tag extraction module is used to extract a set of structured profile tags based on the object's historical structured information; The prompt word construction module is used to obtain a multimodal prompt word set and construct a unified representation sequence based on the data preprocessing results and the multimodal prompt word set; The factor extraction module is used to input the unified representation sequence into the multimodal model to obtain a fusion representation, and extract a candidate set of derived factors from the fusion representation; The consistency verification module is used to perform cross-modal consistency verification on the cross-modal data in the candidate set of derived factors to obtain a consistent factor set; The portrait fusion output module is used to fuse the set of consistent factors with the set of structured portrait tags to generate a unified portrait system and output the set of object portrait tags in the unified portrait system.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and an object image tag set generation program stored in the memory and executable on the processor. When the object image tag set generation program is executed by the processor, it implements the steps of the object image tag set generation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores an object profile tag set generation program, which, when executed by a processor, implements the steps of the object profile tag set generation method as described in any one of claims 1-7.

Images