User file updating method and system for large audio-visual model
By acquiring audio and video streams in real time, extracting facial and voiceprint feature vectors, updating the feature library, and filtering effective dialogue data, the problem of lifelong memory of user information in large models is solved, and the accuracy of user profiles and personalized responses are achieved.
Patent Information
- Application Number
- CN202511537638.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-27
- Publication Date
- 2026-02-10
AI Technical Summary
Existing technologies cannot achieve lifelong memory of user information in large models, especially in embodied intelligent robots and audio and video chat services. They cannot process audiovisual modal information, and existing technologies cannot meet the context window length requirements of multimodal real-time audio and video applications.
By acquiring audio and video streams in real time, extracting facial and voiceprint feature vectors, calculating cosine similarity to update the feature library, and combining sequence labeling models to filter effective dialogue data, converting it into text data to update user profiles, thus achieving lifelong memory of user information.
It enables large models to remember user information for life, improves the accuracy and timeliness of user profiles, ensures the personalization and adaptability of generated responses, and solves the shortcomings of existing technologies that are limited to text-based updates.
Smart Images
Figure CN121502022A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and multimodal interaction technology, and in particular to a method and system for updating user profiles of large audiovisual models. Background Technology
[0002] In large-scale model services based on audiovisual interaction, to achieve personalized responses, the large model needs to remember key information such as each user's identity, social relationships, conversation history, and personal preferences. However, this application faces two major challenges: First, in a lifelong deployment scenario, the model's context cannot be infinitely expanded, posing a challenge to the long-term storage and retrieval of information; second, most current mature technologies are limited to text modalities, making it difficult to handle scenarios requiring audiovisual modal support, such as embodied intelligent robots and audio / video chat services.
[0003] In the current technological context, the main technologies for large model memory and context processing include text-based memory agents and long context window technology, but both have significant shortcomings. While text-based memory agents, through the agent method, enable large text models like GPT-4 and DeepSeek to have lifelong memory capabilities, they are limited to the text modality and cannot process non-textual information such as visual and auditory information. Therefore, they cannot support scenarios requiring audiovisual modalities, such as embodied intelligent robots and audio / video chat services, nor can they determine and switch user identities in real-time based on multimodal information, as required by audiovisual interaction scenarios. Long context window technology typically attempts to contain the entire dialogue history in short-term memory (i.e., the context of the large model's reasoning process), allowing the large model itself to integrate the information. However, its key drawback is the upper limit on the length of the context window, making it difficult to reach the level of millions of input tokens. In multimodal real-time audio / video applications, where the context window length expands rapidly and has no clear upper limit, it cannot meet the requirements for lifelong deployment, presenting a fundamental obstacle at the theoretical level. Summary of the Invention
[0004] The present invention aims to provide a method and system for updating user profiles of large audiovisual models, so as to solve the above-mentioned technical problems, avoid the inability to update user profiles due to being limited to text modality, and realize the lifelong memory function of large models for user information.
[0005] To address the aforementioned technical problems, this invention provides a method for updating user profiles in a large audiovisual model, comprising:
[0006] Real-time acquisition of audio and video streams;
[0007] Based on audio and video streams, extract several facial feature vectors and corresponding voiceprint feature vectors;
[0008] Based on the facial feature vector and the pre-stored facial feature library, the cosine similarity of the face is calculated, and the pre-stored facial feature library is updated based on the facial cosine similarity and the preset facial cosine similarity threshold to obtain the first facial feature library.
[0009] Based on the voiceprint feature vector and the pre-stored voiceprint feature library, the voiceprint cosine similarity is calculated, and the pre-stored voiceprint feature library is updated based on the voiceprint cosine similarity and the preset voiceprint cosine similarity threshold to obtain the first voiceprint feature library.
[0010] The pre-stored user profiles are updated based on the first facial feature database and the first voiceprint feature database to obtain the first user profile;
[0011] Based on the audio and video streams and the preset sequence labeling model, valid dialogue data is obtained and converted into text data;
[0012] The first user profile is updated based on text data and the audio / video stream, and the second user profile is obtained, thereby updating the user profile.
[0013] In the above scheme, the pre-stored face feature library is updated by calculating the cosine similarity of faces and combining it with a preset face cosine similarity threshold to obtain a first face feature library. This achieves dynamic optimization of the face feature library, providing a reliable face feature foundation for subsequent updates to user profiles. Then, the pre-stored voiceprint feature library is updated by calculating the cosine similarity of voiceprints and combining it with a preset voiceprint cosine similarity threshold to obtain a first voiceprint feature library, completing the dynamic maintenance of the voiceprint feature library. Next, the pre-stored user profile is updated using the first face feature library and the first voiceprint feature library, integrating the optimized faces and voiceprints into the user profile. This achieves synchronous updates and improvements to the user profile, ensuring that the generated first user profile accurately reflects the user's current data, supplementing and improving user information, and effectively enhancing the accuracy and timeliness of user data in the user profile. Finally, a preset sequence labeling model is introduced to provide algorithmic support for subsequent filtering of effective dialogue data from audio and video streams, ensuring the reliability of the extracted effective dialogue data. Subsequently, valid dialogue data is extracted from the audio and video streams and converted into text data, providing a basis for updating the first user profile. Finally, the first user profile is updated by combining the text data and the audio and video streams, enriching the user profile dimensions and ultimately obtaining the second user profile. This avoids the inability to update user profiles due to being limited to the text modality, enabling the large model to have a lifelong memory function for user information.
[0014] Furthermore, the step of extracting several facial feature vectors and corresponding voiceprint feature vectors based on audio and video streams includes:
[0015] Extract several image frames based on the audio and video streams;
[0016] Based on the image frame and the pre-trained face feature extraction model, several face feature vectors are obtained;
[0017] Extract several audio segments based on the audio and video streams;
[0018] Based on the audio segment and the pre-trained voiceprint feature extraction model, a voiceprint feature vector is extracted;
[0019] Establish the correspondence between facial feature vectors and voiceprint feature vectors, and obtain the voiceprint feature vectors corresponding to the facial feature vectors.
[0020] In the above scheme, several image frames are extracted from the audio and video stream, providing basic image data for subsequent extraction of facial feature vectors. Next, a pre-trained facial feature extraction model identifies and quantifies key facial features from the image frames, converting the image information into facial feature vectors usable for user identification. Several audio segments are extracted from the audio and video stream, providing basic audio data for subsequent extraction of voiceprint feature vectors. Subsequently, a pre-trained voiceprint feature extraction model captures and quantifies unique voiceprint information from the extracted audio segments, converting the audio information into voiceprint feature vectors usable for user identification. Following this, by establishing a correspondence between facial feature vectors and voiceprint feature vectors, the facial feature vectors and voiceprint feature vectors of the same user are associated, obtaining the voiceprint feature vector corresponding to the facial feature vector, providing a basis for subsequent user profile updates.
[0021] Furthermore, the step of updating the pre-stored user profile based on the first facial feature library and the first voiceprint feature library to obtain the first user profile includes:
[0022] Based on the first facial feature library, retrieve the user information data corresponding to the facial feature vector; based on the first voiceprint feature library, retrieve the user dialogue data corresponding to the voiceprint feature vector.
[0023] The pre-stored user profile is updated based on the user information data corresponding to the facial feature vector and the user dialogue data corresponding to the voiceprint feature vector. The updated pre-stored user profile is then associated with the first stored facial feature library and the first stored voiceprint feature library to obtain the first user profile.
[0024] In the above scheme, user information data is extracted from the updated first facial feature database, and user dialogue data is extracted from the first voiceprint feature database. This provides specific supplementary information for subsequent updates to user profiles, achieving an initial association between feature data and user information. Next, the pre-stored user profiles are updated using the user information data corresponding to the facial feature vectors and the user dialogue data corresponding to the voiceprint feature vectors. The updated pre-stored user profiles are then associated with the first facial feature database and the first voiceprint feature database. This not only supplements and improves the content of the user profiles but also establishes a binding relationship between the user profiles and the latest feature databases. This ensures that the corresponding user profiles can be quickly matched using facial or voiceprint features, improving the relevance and usability of the profiles.
[0025] Furthermore, in obtaining valid dialogue data based on the audio / video stream and the preset sequence labeling model, and converting it into text data, the preset sequence labeling model includes:
[0026] Interrupt and assign tags to the user dialogue data to be acquired, and obtain the tagged dialogue data;
[0027] An initial sequence labeling model is established based on the preset labeling model parameters;
[0028] The initial sequence labeling model is trained based on labeled dialogue data until a preset stopping condition is met, and a preset sequence labeling model is obtained.
[0029] In the above scheme, the pre-acquired user dialogue data is interrupted and labeled to obtain labeled dialogue data. This transforms the raw, unstructured dialogue data into training samples containing labeled information, clarifying the distinction between "valid dialogue" and "invalid content" in the data, and providing input data for subsequent model training. Subsequently, an initial sequence labeling model is established based on preset labeling model parameters, building the basic framework of the model and determining its core structure and initial operating parameters, providing a starting point for subsequent iterative model training. Then, the initial sequence labeling model is trained on the labeled dialogue data, continuously adjusting the model parameters and optimizing the model's ability to identify valid information in the dialogue data until a preset stopping condition is met. This means the model can ultimately stably and accurately filter valid content from the dialogue, obtaining the preset sequence labeling model, and meeting the subsequent requirement of extracting valid dialogue data from audio and video streams.
[0030] Furthermore, the step of acquiring valid dialogue data based on the audio / video stream and a preset sequence labeling model, and converting it into text data, includes:
[0031] Based on the audio segment and the first stored voiceprint feature library, extract the corresponding user dialogue data from the first user profile;
[0032] Extract the time-slice features of the corresponding user dialogue data in the first user profile;
[0033] Based on the time slice features and the preset sequence labeling model, time slice feature labels are obtained;
[0034] If the time slice feature label is within the preset first label and preset second label, then the user dialogue data corresponding to the time slice feature label is determined as valid dialogue data.
[0035] In the above scheme, an updated voiceprint feature library is used to accurately match audio segments with user profiles, extracting corresponding user dialogue data from the first user profile. Next, by extracting time-slice features from the corresponding user dialogue data in the first user profile, continuous user dialogue data is split along the time dimension, and key features are extracted, transforming the dialogue data into structured feature data recognizable by the model, providing a basis for subsequent model annotation. Subsequently, time-slice feature labels are obtained based on the time-slice features and a preset sequence labeling model, and the split dialogue time-slice features are labeled, clarifying the attribute category of each time-slice dialogue content, providing a reference for judging valid dialogue data. Finally, the labeled dialogue time-slices are filtered according to a preset range of valid labels, accurately extracting valid dialogue data that meets the requirements, laying the foundation for subsequent conversion into text data.
[0036] Furthermore, it also includes:
[0037] Retrieve the second user profile based on text data to obtain the second user information data;
[0038] Response data is generated based on the second user information data and text data.
[0039] In the above solution, a second user profile is retrieved through text data to obtain second user information data. This establishes a link between the text data and the user profile, enabling the extraction of user information data relevant to the current conversation from the profile. This provides a personalized basis for generating responses tailored to user needs. Next, response data is generated based on the second user information data and the text data. This user information data is combined with the current conversation content to ensure that the generated response not only addresses the questions or needs in the current text but also incorporates the user's past characteristics, making the response more targeted and personalized, and improving the adaptability of the interaction with the user.
[0040] This invention provides a large-scale audiovisual user profile update system, including an audio and video stream acquisition module, a feature extraction module, a face feature update module, a voiceprint feature update module, a profile update module, a dialogue extraction and translation module, and a profile re-update module, specifically:
[0041] The audio and video stream acquisition module is used to acquire audio and video streams in real time;
[0042] The feature extraction module is used to extract several facial feature vectors and corresponding voiceprint feature vectors based on audio and video streams.
[0043] The face feature update module is used to calculate the face cosine similarity based on the face feature vector and the pre-stored face feature library, and update the pre-stored face feature library based on the face cosine similarity and the preset face cosine similarity threshold to obtain the first face feature library.
[0044] The voiceprint feature update module is used to calculate the voiceprint cosine similarity based on the voiceprint feature vector and the pre-stored voiceprint feature library, and update the pre-stored voiceprint feature library based on the voiceprint cosine similarity and the preset voiceprint cosine similarity threshold to obtain the first voiceprint feature library.
[0045] The file update module is used to update the pre-stored user files based on the first face feature library and the first voiceprint feature library, and to obtain the first user file;
[0046] The dialogue extraction and translation module is used to obtain valid dialogue data based on the audio and video stream and a preset sequence labeling model, and convert it into text data.
[0047] The file update module is used to update the first user file based on text data and the audio / video stream, obtain the second user file, and realize the update of the user file.
[0048] This invention provides an audiovisual large-scale model user profile update system. In practical applications, it only requires a face feature update module. By calculating the cosine similarity of faces and combining it with a preset face cosine similarity threshold, it updates the pre-stored face feature database to obtain a first face feature database, achieving dynamic optimization of the face feature database and providing a reliable face feature foundation for subsequent user profile updates. Then, a voiceprint feature update module is used. By calculating the cosine similarity of voiceprints and combining it with a preset voiceprint cosine similarity threshold, it updates the pre-stored voiceprint feature database to obtain a first voiceprint feature database, completing the dynamic maintenance of the voiceprint feature database. Next, a profile update module is used to update the pre-stored user profile using the first face feature database and the first voiceprint feature database. The optimized face and voiceprint are integrated into the user profile, achieving synchronous updating and improvement of the user profile. This ensures that the generated first user profile accurately reflects the user's current data, supplementing and improving user information, and effectively enhancing the accuracy and timeliness of user data in the user profile. Next, a pre-defined sequence labeling model is introduced to provide algorithmic support for filtering effective dialogue data from audio and video streams, ensuring the reliability of the extracted effective dialogue data. Subsequently, a dialogue extraction and translation module is used to filter effective dialogue data from the audio and video streams and convert it into text data, providing a basis for updating the first user profile. Finally, a profile re-update module is used to update the first user profile by combining text data and audio / video streams, enriching the user profile dimensions and ultimately obtaining the second user profile. This avoids the inability to update user profiles due to being limited to a text modality, enabling the large model to retain user information for life.
[0049] Furthermore, the feature extraction module is used to extract several facial feature vectors and corresponding voiceprint feature vectors based on the audio and video streams; including:
[0050] Extract several image frames based on the audio and video streams;
[0051] Based on the image frame and the pre-trained face feature extraction model, several face feature vectors are obtained;
[0052] Extract several audio segments based on the audio and video streams;
[0053] Based on the audio segment and the pre-trained voiceprint feature extraction model, a voiceprint feature vector is extracted;
[0054] Establish the correspondence between facial feature vectors and voiceprint feature vectors, and obtain the voiceprint feature vectors corresponding to the facial feature vectors.
[0055] In the above scheme, several image frames are extracted from the audio and video stream, providing basic image data for subsequent extraction of facial feature vectors. Next, a pre-trained facial feature extraction model identifies and quantifies key facial features from the image frames, converting the image information into facial feature vectors usable for user identification. Several audio segments are extracted from the audio and video stream, providing basic audio data for subsequent extraction of voiceprint feature vectors. Subsequently, a pre-trained voiceprint feature extraction model captures and quantifies unique voiceprint information from the extracted audio segments, converting the audio information into voiceprint feature vectors usable for user identification. Following this, by establishing a correspondence between facial feature vectors and voiceprint feature vectors, the facial feature vectors and voiceprint feature vectors of the same user are associated, obtaining the voiceprint feature vector corresponding to the facial feature vector, providing a basis for subsequent user profile updates.
[0056] Furthermore, the file update module is used to update the pre-stored user files based on the first facial feature library and the first voiceprint feature library to obtain the first user file; including:
[0057] Based on the first facial feature library, retrieve the user information data corresponding to the facial feature vector; based on the first voiceprint feature library, retrieve the user dialogue data corresponding to the voiceprint feature vector.
[0058] The pre-stored user profile is updated based on the user information data corresponding to the facial feature vector and the user dialogue data corresponding to the voiceprint feature vector. The updated pre-stored user profile is then associated with the first stored facial feature library and the first stored voiceprint feature library to obtain the first user profile.
[0059] In the above scheme, user information data is extracted from the updated first facial feature database, and user dialogue data is extracted from the first voiceprint feature database. This provides specific supplementary information for subsequent updates to user profiles, achieving an initial association between feature data and user information. Next, the pre-stored user profiles are updated using the user information data corresponding to the facial feature vectors and the user dialogue data corresponding to the voiceprint feature vectors. The updated pre-stored user profiles are then associated with the first facial feature database and the first voiceprint feature database. This not only supplements and improves the content of the user profiles but also establishes a binding relationship between the user profiles and the latest feature databases. This ensures that the corresponding user profiles can be quickly matched using facial or voiceprint features, improving the relevance and usability of the profiles.
[0060] Furthermore, in obtaining valid dialogue data based on the audio / video stream and the preset sequence labeling model, and converting it into text data, the preset sequence labeling model includes:
[0061] Interrupt and assign tags to the user dialogue data to be acquired, and obtain the tagged dialogue data;
[0062] An initial sequence labeling model is established based on the preset labeling model parameters;
[0063] The initial sequence labeling model is trained based on labeled dialogue data until a preset stopping condition is met, and a preset sequence labeling model is obtained.
[0064] In the above scheme, the pre-acquired user dialogue data is interrupted and labeled to obtain labeled dialogue data. This transforms the raw, unstructured dialogue data into training samples containing labeled information, clarifying the distinction between "valid dialogue" and "invalid content" in the data, and providing input data for subsequent model training. Subsequently, an initial sequence labeling model is established based on preset labeling model parameters, building the basic framework of the model and determining its core structure and initial operating parameters, providing a starting point for subsequent iterative model training. Then, the initial sequence labeling model is trained on the labeled dialogue data, continuously adjusting the model parameters and optimizing the model's ability to identify valid information in the dialogue data until a preset stopping condition is met. This means the model can ultimately stably and accurately filter valid content from the dialogue, obtaining the preset sequence labeling model, and meeting the subsequent requirement of extracting valid dialogue data from audio and video streams. Attached Figure Description
[0065] Figure 1 A flowchart illustrating a method for updating user profiles in an audiovisual large model according to an embodiment of the present invention;
[0066] Figure 2 This is an architecture diagram of an audiovisual large model user profile update system provided in an embodiment of the present invention. Detailed Implementation
[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0068] This embodiment provides a method for updating user profiles of large-scale audiovisual models. Please refer to the flowchart for details. Figure 1 ,include:
[0069] Step S1: Acquire audio and video streams in real time;
[0070] Step S2: Based on the audio and video streams, extract several facial feature vectors and corresponding voiceprint feature vectors;
[0071] Step S3: Based on the face feature vector and the pre-stored face feature library, calculate the face cosine similarity, and update the pre-stored face feature library based on the face cosine similarity and the preset face cosine similarity threshold to obtain the first face feature library;
[0072] Step S4: Based on the voiceprint feature vector and the pre-stored voiceprint feature library, calculate the voiceprint cosine similarity, and update the pre-stored voiceprint feature library based on the voiceprint cosine similarity and the preset voiceprint cosine similarity threshold to obtain the first voiceprint feature library.
[0073] Step S5: Update the pre-stored user profile based on the first face feature library and the first voiceprint feature library to obtain the first user profile;
[0074] Step S6: Based on the audio and video streams and the preset sequence labeling model, obtain valid dialogue data and convert it into text data;
[0075] Step S7: Update the first user profile based on the text data and the audio / video stream, obtain the second user profile, and realize the update of the user profile.
[0076] In this embodiment, the extracted facial feature vector y is used as the query, and each facial feature vector y' in the pre-stored facial feature database is used as the key. The similarity between the key and query is calculated to obtain several facial cosine similarities. The maximum value is then compared with a preset facial cosine similarity threshold (0.6 in this embodiment). If the value exceeds 0.6, it is identified as the facial feature vector corresponding to y' in the pre-stored facial feature database. Otherwise, it is determined that there is no matching feature vector in the pre-stored facial feature database. For facial feature vectors of new users that fail to match those in the pre-stored facial feature database, the pre-stored facial feature database is updated to obtain a first voiceprint feature database containing the facial feature vector of the new user. This achieves dynamic optimization of the facial feature database, providing a reliable facial feature foundation for subsequent updates to user profiles. Then, using the extracted voiceprint feature vector z as the query and each voiceprint feature vector z' in the pre-stored voiceprint feature library as the key, the similarity between the key and the query is calculated to obtain several voiceprint cosine similarities. The maximum value is then compared with a preset voiceprint cosine similarity threshold. If the value exceeds the preset threshold, it is identified as the voiceprint feature vector corresponding to z' in the pre-stored voiceprint feature library. Otherwise, it is determined that there is no matching feature vector in the pre-stored voiceprint feature library. For the voiceprint feature vector of a new user that fails to match the pre-stored voiceprint feature library, the pre-stored voiceprint feature library is updated to obtain a first voiceprint feature library that stores the voiceprint feature vector of the new user, thus completing the dynamic maintenance of the voiceprint feature library. Since there is no universally applicable preset cosine similarity threshold for voiceprints, it needs to be determined. The specific method is as follows: A pre-stored voiceprint feature library containing 1000 different speakers is constructed. 10,000 (A, A') tuples from this speaker set are used as the validation set, where A and A' in each tuple may come from the same or different speakers. Based on the above steps, the cosine similarity between A and A' is calculated for voiceprint feature verification. Different thresholds are iterated, and the precision and recall of the verification are calculated. When precision equals recall, this threshold is taken as the preset cosine similarity threshold for voiceprints. In practical applications, there may be a trade-off between precision and recall; in this case, the threshold is adjusted upwards or downwards within a 10% range as appropriate. Finally, the pre-stored user profile is updated using the first face feature library and the first voiceprint feature library. The optimized face and voiceprint are integrated into the user profile, achieving synchronous updates and improvements to the user profile, so that the generated first user profile accurately reflects the user's current data. Next, a pre-defined sequence labeling model is introduced to provide algorithmic support for filtering effective dialogue data from audio and video streams, ensuring the reliability of the extracted effective dialogue data.Subsequently, valid dialogue data is filtered from the audio and video streams and converted into text data using Automatic Speech Recognition (ASR) (the ASR model can choose any open-source tool, such as Kaldi, Whisper, etc.), providing a basis for updating the first user profile. Finally, combining the text data and image frames from the audio and video streams, the base model interface is called, and information and summaries are obtained using prompts to update the first user profile, enriching its dimensions and ultimately resulting in the second user profile. This avoids the inability to update user profiles due to limitations in text modality, effectively compensating for the shortcomings of current technologies in multimodal and infinite sequence length scenarios, and realizing the lifelong memory function of large models for user information. A prompt could be exemplified as: "Please extract the user's identity, gender, portrait, and other information based on video frames and dialogue history, and summarize the dialogue content. Example format: [Example of required data format]". The base model can be any model capable of processing both visual and text modalities, such as GPT-4o or Gemini-2.5pro.
[0077] Furthermore, the step of extracting several facial feature vectors and corresponding voiceprint feature vectors based on audio and video streams includes:
[0078] Extract several image frames based on the audio and video streams;
[0079] Based on the image frame and the pre-trained face feature extraction model, several face feature vectors are obtained;
[0080] Extract several audio segments based on the audio and video streams;
[0081] Based on the audio segment and the pre-trained voiceprint feature extraction model, a voiceprint feature vector is extracted;
[0082] Establish the correspondence between facial feature vectors and voiceprint feature vectors, and obtain the voiceprint feature vectors corresponding to the facial feature vectors.
[0083] In this embodiment, one image frame is extracted every 2 seconds from the continuous audio and video stream, resulting in several image frames. The pixel size of each image frame can be arbitrary, providing basic image data for subsequent extraction of facial feature vectors. Next, a pre-trained facial feature extraction model (such as the FaceNet convolutional neural network) identifies and quantifies key facial features from the image frames, converting the image information into a facial feature vector usable for user recognition, i.e., y = F(I), where F represents the parameters of the convolutional neural network (using a publicly available version, requiring no further training), and I represents the input image frame. The FaceNet convolutional neural network performs feedforward operations to extract the facial feature vector y, which is a 512-dimensional feature vector. The most recent 5-second audio clip is extracted every 2 seconds from the continuous audio and video stream, obtaining several audio segments. These audio segments are represented using continuous temporal waveforms, providing basic audio data for subsequent extraction of voiceprint feature vectors. Subsequently, a pre-trained voiceprint feature extraction model (such as the WavLM model) captures and quantifies unique voiceprint information from the extracted audio segments, transforming the audio information into a voiceprint feature vector that can be used for user identification, i.e., z = W(A), where W is the network parameter of the WavLM model, and A is the input audio segment. The WavLM model performs feedforward operations to extract the voiceprint feature vector z. Next, by establishing a correspondence between facial feature vectors and voiceprint feature vectors, the facial feature vectors and voiceprint feature vectors of the same user are associated to obtain the voiceprint feature vector corresponding to the facial feature vector, providing a basis for subsequent user profile updates.
[0084] Furthermore, the step of updating the pre-stored user profile based on the first facial feature library and the first voiceprint feature library to obtain the first user profile includes:
[0085] Based on the first facial feature library, retrieve the user information data corresponding to the facial feature vector; based on the first voiceprint feature library, retrieve the user dialogue data corresponding to the voiceprint feature vector.
[0086] The pre-stored user profile is updated based on the user information data corresponding to the facial feature vector and the user dialogue data corresponding to the voiceprint feature vector. The updated pre-stored user profile is then associated with the first stored facial feature library and the first stored voiceprint feature library to obtain the first user profile.
[0087] In this embodiment, user information data is extracted from the updated first facial feature database, and user dialogue data is extracted from the first voiceprint feature database. This provides specific supplementary information for subsequent updates to user profiles, achieving an initial association between feature data and user information. Next, the pre-stored user profiles are updated using the user information data corresponding to the facial feature vectors and the user dialogue data corresponding to the voiceprint feature vectors. The updated pre-stored user profiles are then associated with the first facial feature database and the first voiceprint feature database. This not only supplements and improves the content of the user profiles but also establishes a binding relationship between the user profiles and the latest feature databases. This ensures that the corresponding user profiles can be quickly matched using facial or voiceprint features, improving the relevance and usability of the profiles.
[0088] Furthermore, in obtaining valid dialogue data based on the audio / video stream and the preset sequence labeling model, and converting it into text data, the preset sequence labeling model includes:
[0089] Interrupt and assign tags to the user dialogue data to be acquired, and obtain the tagged dialogue data;
[0090] An initial sequence labeling model is established based on the preset labeling model parameters;
[0091] The initial sequence labeling model is trained based on labeled dialogue data until a preset stopping condition is met, and a preset sequence labeling model is obtained.
[0092] In this embodiment, pre-acquired user dialogue data is used as a draft. If the pre-acquired user dialogue data is a text draft, audio question-and-answer and dialogue are generated through text-to-speech (TTS). The audio question-and-answer and dialogue are then spliced, interrupted, and interleaved. Labels are assigned based on the correct speaker and dialogue occurrence time to obtain labeled dialogue data. This transforms the raw, unstructured dialogue data into training samples containing labeled information, clarifying the distinction between "valid dialogue" and "invalid content" in the data, and providing input data for subsequent model training. Subsequently, an initial sequence labeling model (using a Llama architecture Transformer model with 100 million parameters and bidirectional attention masks, meaning each time-slice feature fully attends to all other time-slice features) is established based on preset labeled model parameters (randomly set). This builds the basic framework of the model, determines its core structure and initial operating parameters (setting hyperparameters, such as a batch size of 64 and a learning rate of 1e-4 decaying cosinely), providing a starting point for subsequent iterative model training, with a preset total of 15 training rounds. Then, the initial sequence labeling model is trained based on labeled dialogue data. The model parameters are continuously adjusted to optimize the model's ability to recognize effective information in the dialogue data until a preset stopping condition is reached. This means the model can consistently and accurately filter effective content from the dialogue, obtaining the preset sequence labeling model to meet the subsequent requirement of extracting effective dialogue data from audio and video streams. The preset stopping condition is as follows: 1% of the pre-acquired user dialogue data is taken as a validation set. Gradient descent is performed every 1000 steps, and the F1 score for each label on the validation set is calculated. The F1 score is calculated as follows: Accuracy P = Number of dialogue data points actually labeled by the model / Total number of dialogue data points labeled by the model; Recall R = Number of dialogue data points actually labeled by the model / Total number of dialogue data points actually labeled by the model; Calculation... F1 represents the harmonic mean, and the preset stopping condition is that training ends when F1 converges or stops increasing. In this embodiment, the preset sequence labeling model can also be replaced by a rule-based model based on silence / volume perception and real-time speaker matching, but this will sacrifice accuracy and introduce error propagation compared to the above methods.
[0093] Furthermore, the step of acquiring valid dialogue data based on the audio / video stream and a preset sequence labeling model, and converting it into text data, includes:
[0094] Based on the audio segment and the first stored voiceprint feature library, extract the corresponding user dialogue data from the first user profile;
[0095] Extract the time-slice features of the corresponding user dialogue data in the first user profile;
[0096] Based on the time slice features and the preset sequence labeling model, time slice feature labels are obtained;
[0097] If the time slice feature label is within the preset first label and preset second label, then the user dialogue data corresponding to the time slice feature label is determined as valid dialogue data.
[0098] In this embodiment, an updated voiceprint feature library is used to accurately match audio segments with user profiles, extracting corresponding user dialogue data from the first user profile. Next, an audio encoder extracts time-slice features from the corresponding user dialogue data in the first user profile, extracting an average of N frames of features per second. Each feature is called a time slice, thus splitting continuous user dialogue data along the time dimension and extracting key features, transforming the dialogue data into structured feature data recognizable by the model, providing a basis for subsequent model annotation. Subsequently, time-slice feature labels are obtained based on the time-slice features and an offline preset sequence labeling model, and the split dialogue time-slice features are labeled. The label system is: {0: no dialogue occurred; 1: the start position of a new user's dialogue; 2: dialogue is in progress; 3: the end position of the dialogue}, clarifying the attribute category of the dialogue content in each time slice, providing a reference for judging valid dialogue data. Finally, the labeled dialogue time slices are filtered according to a preset valid label range (all time slices labeled between 1 and 3), accurately extracting valid dialogue data that meets the requirements, laying the foundation for subsequent conversion into text data.
[0099] Furthermore, it also includes:
[0100] Retrieve the second user profile based on text data to obtain the second user information data;
[0101] Response data is generated based on the second user information data and text data.
[0102] In this embodiment, a second user profile is retrieved via text data to obtain second user information data, establishing a link between the text data and the user profile. This allows for the extraction of user information data relevant to the current conversation from the profile, providing a personalized basis for generating responses tailored to user needs. Next, response data is generated based on the second user information data and the text data. This user information data is combined with the current conversation content to ensure that the generated response not only addresses the questions or needs in the current text but also incorporates the user's past characteristics, making the response more targeted and personalized, and improving the adaptability of the interaction with the user. Finally, a text-to-speech (TTS) model can be used to convert the response data into speech output, realizing voice dialogue. Any open-source model such as Qwen-Audio or FishSpeech can be used for TTS.
[0103] Please see Figure 2This embodiment provides an audiovisual large-scale user profile update system, including an audio and video stream acquisition module, a feature extraction module, a face feature update module, a voiceprint feature update module, a profile update module, a dialogue extraction and translation module, and a profile re-update module, specifically:
[0104] The audio and video stream acquisition module is used to acquire audio and video streams in real time;
[0105] The feature extraction module is used to extract several facial feature vectors and corresponding voiceprint feature vectors based on audio and video streams.
[0106] The face feature update module is used to calculate the face cosine similarity based on the face feature vector and the pre-stored face feature library, and update the pre-stored face feature library based on the face cosine similarity and the preset face cosine similarity threshold to obtain the first face feature library.
[0107] The voiceprint feature update module is used to calculate the voiceprint cosine similarity based on the voiceprint feature vector and the pre-stored voiceprint feature library, and update the pre-stored voiceprint feature library based on the voiceprint cosine similarity and the preset voiceprint cosine similarity threshold to obtain the first voiceprint feature library.
[0108] The file update module is used to update the pre-stored user files based on the first face feature library and the first voiceprint feature library, and to obtain the first user file;
[0109] The dialogue extraction and translation module is used to obtain valid dialogue data based on the audio and video stream and a preset sequence labeling model, and convert it into text data.
[0110] The file update module is used to update the first user file based on text data and the audio / video stream, obtain the second user file, and realize the update of the user file.
[0111] This embodiment provides an audiovisual large-scale model user profile update system. In practical applications, it only requires a face feature update module. The system uses the extracted face feature vector y as a query and each face feature vector y' in the pre-stored face feature database as a key. Similarity is calculated between the key and query to obtain several face cosine similarities. The maximum value is compared with a preset face cosine similarity threshold (0.6 in this embodiment). If the value exceeds 0.6, it is identified as the face feature vector corresponding to y' in the pre-stored face feature database. Otherwise, it is determined that there is no matching feature vector in the pre-stored face feature database. For the face feature vector of a new user that does not match the pre-stored face feature database, the pre-stored face feature database is updated to obtain a first voiceprint feature database containing the face feature vector of the new user. This achieves dynamic optimization of the face feature database, providing a reliable face feature foundation for subsequent user profile updates. Then, a voiceprint feature update module is used. The extracted voiceprint feature vector z is used as the query, and each voiceprint feature vector z' in the pre-stored voiceprint feature library is used as the key. Similarity is calculated between the key and the query to obtain several voiceprint cosine similarities. The maximum value is compared with a preset voiceprint cosine similarity threshold. If it exceeds the preset threshold, it is identified as the voiceprint feature vector corresponding to z' in the pre-stored voiceprint feature library. Otherwise, it is determined that there is no matching feature vector in the pre-stored voiceprint feature library. For voiceprint feature vectors of new users that do not match in the pre-stored voiceprint feature library, the pre-stored voiceprint feature library is updated to obtain a first voiceprint feature library that stores the voiceprint feature vector of the new user, thus completing the dynamic maintenance of the voiceprint feature library. Since there is no universally applicable preset voiceprint cosine similarity threshold, it needs to be determined. The specific method is as follows: A pre-stored voiceprint feature library containing 1000 different speakers is constructed. 10,000 (A, A') tuples from this speaker set are used as the validation set, where A and A' in each tuple may come from the same or different speakers. Based on the above steps, the cosine similarity between A and A' is calculated for voiceprint feature verification. Different thresholds are iterated, and the accuracy and recall of the verification are calculated. When accuracy equals recall, this threshold is taken as the preset voiceprint cosine similarity threshold. In practical applications, there may be a trade-off between accuracy and recall; in this case, the threshold is adjusted upwards or downwards within a 10% range depending on the situation. Next, a profile update module is used to update the pre-stored user profiles using the first face feature library and the first voiceprint feature library. The optimized face and voiceprint are integrated into the user profile, achieving synchronous updates and improvements to the user profile, so that the generated first user profile accurately reflects the user's current data.Next, a dialogue extraction and translation module is employed, introducing a pre-defined sequence labeling model to provide algorithmic support for subsequent filtering of effective dialogue data from the audio and video streams, ensuring the reliability of the extracted effective dialogue data. Subsequently, effective dialogue data is filtered from the audio and video streams and converted into text data through Automatic Speech Recognition (ASR) (the ASR model can be any open-source tool, such as Kaldi, Whisper, etc.), providing a basis for updating the first user profile. Finally, a profile re-update module is used, combining text data and image frames from the audio and video streams, calling the base model interface and utilizing prompts to obtain information and summaries, updating the first user profile, enriching the user profile dimensions, and ultimately obtaining the second user profile. This avoids the inability to update user profiles due to limitations in text modality, effectively compensating for the shortcomings of current technologies in multimodal and infinite sequence length scenarios, and realizing the lifelong memory function of user information for large models. A prompt could be exemplified as: "Please extract the user's identity, gender, portrait, and other information based on video frames and dialogue history, and summarize the dialogue content. Example format: [Example of required data format]". The pedestal model can be any model capable of handling both visual and textual modalities, such as GPT-4o or Gemini-2.5pro.
[0112] Furthermore, the feature extraction module is used to extract several facial feature vectors and corresponding voiceprint feature vectors based on the audio and video streams; including:
[0113] Extract several image frames based on the audio and video streams;
[0114] Based on the image frame and the pre-trained face feature extraction model, several face feature vectors are obtained;
[0115] Extract several audio segments based on the audio and video streams;
[0116] Based on the audio segment and the pre-trained voiceprint feature extraction model, a voiceprint feature vector is extracted;
[0117] Establish the correspondence between facial feature vectors and voiceprint feature vectors, and obtain the voiceprint feature vectors corresponding to the facial feature vectors.
[0118] In this embodiment, one image frame is extracted every 2 seconds from the continuous audio and video stream, resulting in several image frames. The pixel size of each image frame can be arbitrary, providing basic image data for subsequent extraction of facial feature vectors. Next, a pre-trained facial feature extraction model (such as the FaceNet convolutional neural network) identifies and quantifies key facial features from the image frames, converting the image information into a facial feature vector usable for user recognition, i.e., y = F(I), where F represents the parameters of the convolutional neural network (using a publicly available version, requiring no further training), and I represents the input image frame. The FaceNet convolutional neural network performs feedforward operations to extract the facial feature vector y, which is a 512-dimensional feature vector. The most recent 5-second audio clip is extracted every 2 seconds from the continuous audio and video stream, obtaining several audio segments. These audio segments are represented using continuous temporal waveforms, providing basic audio data for subsequent extraction of voiceprint feature vectors. Subsequently, a pre-trained voiceprint feature extraction model (such as the WavLM model) captures and quantifies unique voiceprint information from the extracted audio segments, transforming the audio information into a voiceprint feature vector that can be used for user identification, i.e., z = W(A), where W is the network parameter of the WavLM model, and A is the input audio segment. The WavLM model performs feedforward operations to extract the voiceprint feature vector z. Next, by establishing a correspondence between facial feature vectors and voiceprint feature vectors, the facial feature vectors and voiceprint feature vectors of the same user are associated to obtain the voiceprint feature vector corresponding to the facial feature vector, providing a basis for subsequent user profile updates.
[0119] Furthermore, the file update module is used to update the pre-stored user files based on the first facial feature library and the first voiceprint feature library to obtain the first user file; including:
[0120] Based on the first facial feature library, retrieve the user information data corresponding to the facial feature vector; based on the first voiceprint feature library, retrieve the user dialogue data corresponding to the voiceprint feature vector.
[0121] The pre-stored user profile is updated based on the user information data corresponding to the facial feature vector and the user dialogue data corresponding to the voiceprint feature vector. The updated pre-stored user profile is then associated with the first stored facial feature library and the first stored voiceprint feature library to obtain the first user profile.
[0122] In this embodiment, user information data is extracted from the updated first facial feature database, and user dialogue data is extracted from the first voiceprint feature database. This provides specific supplementary information for subsequent updates to user profiles, achieving an initial association between feature data and user information. Next, the pre-stored user profiles are updated using the user information data corresponding to the facial feature vectors and the user dialogue data corresponding to the voiceprint feature vectors. The updated pre-stored user profiles are then associated with the first facial feature database and the first voiceprint feature database. This not only supplements and improves the content of the user profiles but also establishes a binding relationship between the user profiles and the latest feature databases. This ensures that the corresponding user profiles can be quickly matched using facial or voiceprint features, improving the relevance and usability of the profiles.
[0123] Furthermore, in obtaining valid dialogue data based on the audio / video stream and the preset sequence labeling model, and converting it into text data, the preset sequence labeling model includes:
[0124] Interrupt and assign tags to the user dialogue data to be acquired, and obtain the tagged dialogue data;
[0125] An initial sequence labeling model is established based on the preset labeling model parameters;
[0126] The initial sequence labeling model is trained based on labeled dialogue data until a preset stopping condition is met, and a preset sequence labeling model is obtained.
[0127] In this embodiment, pre-acquired user dialogue data is used as a draft. If the pre-acquired user dialogue data is a text draft, audio question-and-answer and dialogue are generated through text-to-speech (TTS). The audio question-and-answer and dialogue are then spliced, interrupted, and interleaved. Labels are assigned based on the correct speaker and dialogue occurrence time to obtain labeled dialogue data. This transforms the raw, unstructured dialogue data into training samples containing labeled information, clarifying the distinction between "valid dialogue" and "invalid content" in the data, and providing input data for subsequent model training. Subsequently, an initial sequence labeling model (using a Llama architecture Transformer model with 100 million parameters and bidirectional attention masks, meaning each time-slice feature fully attends to all other time-slice features) is established based on preset labeled model parameters (randomly set). This builds the basic framework of the model, determines its core structure and initial operating parameters (setting hyperparameters, such as a batch size of 64 and a learning rate of 1e-4 decaying cosinely), providing a starting point for subsequent iterative model training, with a preset total of 15 training rounds. Then, the initial sequence labeling model is trained based on labeled dialogue data. The model parameters are continuously adjusted to optimize the model's ability to recognize effective information in the dialogue data until a preset stopping condition is reached. This means the model can consistently and accurately filter effective content from the dialogue, obtaining the preset sequence labeling model to meet the subsequent requirement of extracting effective dialogue data from audio and video streams. The preset stopping condition is as follows: 1% of the pre-acquired user dialogue data is taken as a validation set. Gradient descent is performed every 1000 steps, and the F1 score for each label on the validation set is calculated. The F1 score is calculated as follows: Accuracy P = Number of dialogue data points actually labeled by the model / Total number of dialogue data points labeled by the model; Recall R = Number of dialogue data points actually labeled by the model / Total number of dialogue data points actually labeled by the model; Calculation... F1 represents the harmonic mean, and the preset stopping condition is that training ends when F1 converges or stops increasing. In this embodiment, the preset sequence labeling model can also be replaced by a rule-based model based on silence / volume perception and real-time speaker matching, but this will sacrifice accuracy and introduce error propagation compared to the above methods.
[0128] In this embodiment, it is not necessary to perform Automatic Speech Recognition (ASR) on the valid dialogue data in the audio and video streams. To enable the native real-time multimodal dialogue model to have personalized dialogue capabilities, a large number of tuples (including user dialogue data, user information data, and response data) generated by the user profile system proposed in this embodiment can be used as training data to fine-tune the native real-time multimodal dialogue model, thereby obtaining an end-to-end personalized dialogue model. The specific process is as follows: First, collect text dialogue data, each data point containing several (user dialogue data Q, response data Re) tuples, or (audio / video stream I, user dialogue data Q, response data Re) triples. Next, extract user information data and user dialogue data from the above text dialogue data. Subsequently, combine the user information data and user dialogue data with several random rounds of (Q, Re) or (I, Q, Re) data. Then, based on the above data combination, call the external model interface to generate personalized dialogue. The prompt is as follows: "Please generate a personalized dialogue with similar content based on the following dialogue. Note that you should combine user information data and user dialogue data to improve the model's response, making it more consistent with the user profile and needs reflected in the current user profile. Original dialogue: [Q,Re / I,Q,Re], user information data and user dialogue data: [S]". The external model interface returns personalized response data Re'. Further, based on each (Q,Re',S) or (I,Q,Re',S), text-to-speech (TTS) is performed on Q and Re', adding a certain proportion of interruptions, noise, and aliasing to obtain a real-time voice dialogue training dataset D. D is split into a training set and a validation set (99:1 sample ratio). The real-time native voice dialogue model is trained based on the training set. The model structure and parameter count can be arbitrary, referencing Qwen2.5-omni, Moshi, RoboEgo, etc. The model can be designed based on the above structure and randomly initialized, or it can load publicly available pre-trained parameters and fine-tune on the training set of D. Hyperparameters are set as follows: batch size is 256, learning rate is 1e-5, decaying in a cosine manner. The preset total number of training epochs is 5. During training, the quality of the model's personalized dialogue generation is automatically and objectively evaluated based on the validation set. The evaluation dimensions are: the degree of grasp of user information data and user profiles (i.e., factual accuracy, 0%–100%), and dialogue quality score (0–10). Factual accuracy can be calculated as follows: (Number of personalized response data in the personalized response data that mentions content relevant to user dialogue data, consistent with user information data, and consistent with user profiles / Total number of personalized response data mentioning user facts) * 100%.The dialogue quality score removes user profile factors, uses a preset reference answer, and scores the quality and reasonableness of the model's personalized responses in each round based solely on this preset reference answer. The average of the model's responses across all dialogue rounds is then calculated. Training stops when the scores for both automated objective evaluation dimensions cease to increase. The scores for the two automated objective evaluation dimensions can be obtained by calling external large text models (such as GPT-4 or DeepSeek-V3) to obtain factual accuracy and dialogue quality scores. The example prompt is: "The following is a personalized dialogue ability scoring task, where the model needs to provide correct and personalized responses based on the user's identity facts. For each round of dialogue, you need to provide a factual accuracy score and a dialogue quality score. The factual accuracy score is scored as follows: if the model's response appropriately mentions the user's name, dialogue history, or user profile, and matches the facts stored in the user profile provided before the dialogue, it receives 1 point; otherwise, it receives 0 points. If the model's response does not involve the user's name, dialogue history, or user profile, it is marked as 'irrelevant.' The dialogue quality score is scored as follows: only the parts relevant to the question itself are evaluated, without considering the parts related to the user's name, dialogue history, or user profile. A default reference standard answer is provided: if the model's answer to the question itself is reasonable, it should receive 8 points or higher; otherwise, it should not exceed 8 points. The maximum score is 10 points." This allows the external text model to think first and then organize the score using a piece of JSON code at the end of the response, with the round number as the key and [factual accuracy score, dialogue quality score] as the value. The output includes the scores for all rounds of dialogue.
[0129] This embodiment can be applied to welcoming robots. In open scenarios, welcoming robots equipped with the user profile system of this embodiment can directly extract and create information about new users through video and audio dialogue, and continuously and successfully identify the user and retrieve their profile information in subsequent services, rather than relying on user pre-login for profile creation. This enables the robot to correctly guide guests and provide personalized services through each guest's user profile.
[0130] This embodiment can also be applied to conference service robots. In conference service scenarios, multiple users with known identities are involved, and the robot receives instructions such as "pour a glass of water for the technical lead" or "call my assistant over." The conference service robot equipped with the user profile system of this embodiment can not only remember each user's profile for life, but also dynamically update and maintain information such as each person's position and relationships. At the same time, the model's response to instructions is real-time, ensuring execution speed.
[0131] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A method for updating user profiles in a large-scale audiovisual model, characterized in that, include: Real-time acquisition of audio and video streams; Based on audio and video streams, extract several facial feature vectors and corresponding voiceprint feature vectors; Based on the facial feature vector and the pre-stored facial feature library, the cosine similarity of the face is calculated, and the pre-stored facial feature library is updated based on the facial cosine similarity and the preset facial cosine similarity threshold to obtain the first facial feature library. Based on the voiceprint feature vector and the pre-stored voiceprint feature library, the voiceprint cosine similarity is calculated, and the pre-stored voiceprint feature library is updated based on the voiceprint cosine similarity and the preset voiceprint cosine similarity threshold to obtain the first voiceprint feature library. The pre-stored user profiles are updated based on the first facial feature database and the first voiceprint feature database to obtain the first user profile; Based on the audio and video streams and the preset sequence labeling model, valid dialogue data is obtained and converted into text data; The first user profile is updated based on text data and the audio / video stream, and the second user profile is obtained, thereby updating the user profile.
2. The method for updating user profiles of an audiovisual large model according to claim 1, characterized in that, The step involves extracting several facial feature vectors and corresponding voiceprint feature vectors based on audio and video streams; including: Extract several image frames based on the audio and video streams; Based on the image frame and the pre-trained face feature extraction model, several face feature vectors are obtained; Extract several audio segments based on the audio and video streams; Based on the audio segment and the pre-trained voiceprint feature extraction model, a voiceprint feature vector is extracted; Establish the correspondence between facial feature vectors and voiceprint feature vectors, and obtain the voiceprint feature vectors corresponding to the facial feature vectors.
3. The method for updating user profiles of a large audiovisual model according to claim 1, characterized in that, The step of updating the pre-stored user profile based on the first facial feature database and the first voiceprint feature database to obtain the first user profile includes: Based on the first facial feature library, retrieve the user information data corresponding to the facial feature vector; based on the first voiceprint feature library, retrieve the user dialogue data corresponding to the voiceprint feature vector. The pre-stored user profile is updated based on the user information data corresponding to the facial feature vector and the user dialogue data corresponding to the voiceprint feature vector. The updated pre-stored user profile is then associated with the first stored facial feature library and the first stored voiceprint feature library to obtain the first user profile.
4. The method for updating user profiles of a large audiovisual model according to claim 1, characterized in that, In the process of acquiring valid dialogue data and converting it into text data based on the audio / video stream and a preset sequence labeling model, the preset sequence labeling model includes: Interrupt and assign tags to the user dialogue data to be acquired, and obtain the tagged dialogue data; An initial sequence labeling model is established based on the preset labeling model parameters; The initial sequence labeling model is trained based on labeled dialogue data until a preset stopping condition is met, and a preset sequence labeling model is obtained.
5. The method for updating user profiles of a large audiovisual model according to claim 2, characterized in that, The process of acquiring valid dialogue data based on the audio / video stream and a preset sequence labeling model, and converting it into text data, includes: Based on the audio segment and the first stored voiceprint feature library, extract the corresponding user dialogue data from the first user profile; Extract the time-slice features of the corresponding user dialogue data in the first user profile; Based on the time slice features and the preset sequence labeling model, time slice feature labels are obtained; If the time slice feature label is within the preset first label and preset second label, then the user dialogue data corresponding to the time slice feature label is determined as valid dialogue data.
6. The method for updating user profiles of an audiovisual large model according to claim 1, characterized in that, Also includes: Retrieve the second user profile based on text data to obtain the second user information data; Response data is generated based on the second user information data and text data.
7. A user profile update system for a large-scale audiovisual model, characterized in that, It includes an audio / video stream acquisition module, a feature extraction module, a face feature update module, a voiceprint feature update module, a file update module, a dialogue extraction and translation module, and a file re-update module, specifically: The audio and video stream acquisition module is used to acquire audio and video streams in real time; The feature extraction module is used to extract several facial feature vectors and corresponding voiceprint feature vectors based on audio and video streams. The face feature update module is used to calculate the face cosine similarity based on the face feature vector and the pre-stored face feature library, and update the pre-stored face feature library based on the face cosine similarity and the preset face cosine similarity threshold to obtain the first face feature library. The voiceprint feature update module is used to calculate the voiceprint cosine similarity based on the voiceprint feature vector and the pre-stored voiceprint feature library, and update the pre-stored voiceprint feature library based on the voiceprint cosine similarity and the preset voiceprint cosine similarity threshold to obtain the first voiceprint feature library. The file update module is used to update the pre-stored user files based on the first face feature library and the first voiceprint feature library, and to obtain the first user file; The dialogue extraction and translation module is used to obtain valid dialogue data based on the audio and video stream and a preset sequence labeling model, and convert it into text data. The file update module is used to update the first user file based on text data and the audio / video stream, obtain the second user file, and realize the update of the user file.
8. The audiovisual large-scale model user profile update system according to claim 7, characterized in that, The feature extraction module is used to extract several facial feature vectors and corresponding voiceprint feature vectors based on the audio and video streams; including: Extract several image frames based on the audio and video streams; Based on the image frame and the pre-trained face feature extraction model, several face feature vectors are obtained; Extract several audio segments based on the audio and video streams; Based on the audio segment and the pre-trained voiceprint feature extraction model, a voiceprint feature vector is extracted; Establish the correspondence between facial feature vectors and voiceprint feature vectors, and obtain the voiceprint feature vectors corresponding to the facial feature vectors.
9. The audiovisual large-scale model user profile update system according to claim 7, characterized in that, The file update module is used to update the pre-stored user files based on the first facial feature library and the first voiceprint feature library, and to obtain the first user file; including: Based on the first facial feature library, retrieve the user information data corresponding to the facial feature vector; based on the first voiceprint feature library, retrieve the user dialogue data corresponding to the voiceprint feature vector. The pre-stored user profile is updated based on the user information data corresponding to the facial feature vector and the user dialogue data corresponding to the voiceprint feature vector. The updated pre-stored user profile is then associated with the first stored facial feature library and the first stored voiceprint feature library to obtain the first user profile.
10. The audiovisual large-scale model user profile update system according to claim 7, characterized in that, In the process of acquiring valid dialogue data and converting it into text data based on the audio / video stream and a preset sequence labeling model, the preset sequence labeling model includes: Interrupt and assign tags to the user dialogue data to be acquired, and obtain the tagged dialogue data; An initial sequence labeling model is established based on the preset labeling model parameters; The initial sequence labeling model is trained based on labeled dialogue data until a preset stopping condition is met, and a preset sequence labeling model is obtained.