Scene-adaptive digital human behavior prediction method and device, and storage medium

Abstract

The invention provides scene-adaptive digital human behavior prediction equipment and a storage medium, and the method comprises the steps: collecting multi-modal user interaction data such as texts, voices, images and the like, and carrying out the preprocessing; the method comprises the following steps: extracting a multi-modal feature vector through a pre-trained multi-modal feature extraction model; and then, predicting behaviors of the digital person in different scenes by using a pre-trained scene recognition model and a behavior prediction model. In order to improve the prediction accuracy, a transfer learning technology is adopted, and model parameters or structures are optimized according to historical and current data differences. Meanwhile, user feedback is collected in real time, and the prediction model is iteratively updated to meet the requirements of different user groups. The adaptive method not only improves the accuracy of digital human behavior prediction, but also enhances the understanding and adaptive capabilities of the digital human behavior prediction to different scenes, thereby providing more intelligent and humanized services.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a scene-adaptive digital human behavior prediction method, device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, digital human technology has become a hot topic in current research. Digital humans are virtual characters that simulate human characteristics and exist in the non-physical world, built with the help of digital technology. They use technologies such as natural language processing, face recognition, and voice recognition to achieve intelligent interaction with users, and are widely used in many fields such as virtual anchor customer service and online education.

[0003] At present, the existing digital human behavior prediction technologies mainly include rule-based models and traditional machine learning models. Rule-based models often require a large number of rules to be set manually. For example, in the virtual anchor scenario, response rules are set for different types of audience questions. However, when faced with complex and changeable user behaviors, this method has low prediction accuracy because the rules are difficult to enumerate. Traditional machine learning models, such as decision tree models, may have deviations in predicting digital human behavior in customer service scenarios due to insufficient understanding of the special needs of different customer groups. Moreover, most of these existing technologies are developed based on a single scenario or simple behavior pattern, lacking in-depth understanding and adaptability to different scenarios. This leads to low accuracy in digital human behavior prediction in practical applications, making it difficult to provide intelligent and humanized services. Summary of the invention

[0004] The present invention aims to at least solve the technical problem of low prediction accuracy in the prior art, and in particular innovatively proposes a scene-adaptive digital human behavior prediction method, device and storage medium.

[0005] In order to achieve the above-mentioned object of the present invention, the present invention provides a scene-adaptive digital human behavior prediction method, the method comprising:

[0006] S1. Collecting multimodal user interaction data and preprocessing the multimodal user interaction data; the multimodal user interaction data includes text data, voice data and image data;

[0007] S2. Acquire a multimodal feature vector in the multimodal user interaction data based on a pre-trained multimodal feature extraction model;

[0008] S3, obtaining a scene recognition result using a pre-trained scene recognition model based on the multimodal feature vector, and predicting the behavior of the digital human using a behavior prediction model based on the scene recognition result to obtain a digital human behavior prediction result;

[0009] S4. Based on the digital human behavior prediction results, a transfer learning mechanism is introduced to optimize the behavior prediction model:

[0010] S5. Collect user feedback information in real time, and iteratively update the prediction model based on the user feedback information.

[0011] As an optional embodiment of the present invention, optionally, obtaining the multimodal feature vector in the multimodal user interaction data in step S2 includes:

[0012] S101, extracting semantic features from the text data based on a pre-trained BERT model;

[0013] S102, converting the speech data into text based on a speech recognition model, and extracting emotional features;

[0014] S103, extracting visual features from the image data based on a face recognition model;

[0015] S104, fusing the semantic features, taking the emotional features and the visual features based on the attention mechanism to obtain a multimodal feature vector.

[0016] As an optional embodiment of the present invention, optionally, the expression for obtaining the multimodal feature vector in step S104 is:

[0017]

[0018]

[0019] in, represents the mapped semantic feature vector, W T represents the weight matrix of semantic features, T represents semantic features, b T The bias vector representing the semantic feature, represents the mapped sentiment feature vector, W A Represents the weight matrix of emotional features, A represents emotional features, b A The bias vector representing the sentiment feature, represents the mapped visual feature vector, W V represents the weight matrix of visual features, V represents visual features, b V Represents the bias vector of visual features, α T represents the attention weight of the semantic feature, exp( ) represents the exponential function, and Both represent weight vectors, b′ T , b′ A and b′ V express and The corresponding bias term, F represents the fused multimodal feature vector.

[0020] As an optional embodiment of the present invention, optionally, obtaining the digital human behavior prediction result in step S3 includes:

[0021] S301, collecting historical behavior data of digital humans, and sorting the historical behavior data according to time series to obtain a time series data set;

[0022] S302, extracting digital human behavior-related features from the time series data set;

[0023] S303, obtaining the digital human behavior trend based on the behavior-related characteristics;

[0024] S304, based on the behavior trend and scene recognition results, using the trained behavior prediction model to predict the behavior of the digital human, and obtain a prediction probability distribution of the digital human behavior;

[0025] S305: Based on the digital human behavior prediction probability distribution, select the behavior with the highest probability as the digital human behavior prediction result.

[0026] As an optional embodiment of the present invention, optionally, the expression for obtaining the behavior trend of the digital human based on the behavior-related features is:

[0027] D t =η·M t +β·S t +γ·E t

[0028] M t =λ·(Y t -Y t-1 )+(1-λ)·M t-1

[0029]

[0030]

[0031] Among them, D t represents the behavior trend, η, β and γ represent the average weight coefficients, M t represents the momentum of the behavior change at time t, S t Indicates seasonal changes in behavior, E t represents the impact of external factors on behavior, λ represents the momentum smoothing factor, and Y t represents the behavior label of the digital human at time t, Y t-1 The digital human behavior label at time t-1, M t-1The momentum of the behavior change at time t-1, K represents the length of the seasonal cycle, φ k represents the seasonal weight coefficient, Y t-k·P represents the behavior label at time tk·P, P represents the basic unit of seasonal cycle, N represents the number of external influencing factors, θ i represents the weight coefficient of the i-th external influencing factor, E i,t Represents the value of the i-th external influencing factor at time t.

[0032] As an optional embodiment of the present invention, optionally, in step S04, the expression for obtaining the prediction probability distribution of digital human behavior is:

[0033] P t =f(D t ,H t )

[0034] P t =softmax(W o ·h t +b o )

[0035] h t =σ(W h ·[D t ;H t ]+b h )

[0036] Among them, P t represents the probability distribution of various behaviors that the digital human may take at time t, f( ) represents the behavior prediction model, and D t Indicates behavioral trend, H t represents the scene recognition result at time t, softmax() represents the activation function, W o represents the weight matrix of the output layer, h t represents the hidden layer output at time t, b o represents the bias vector of the output layer, σ( ) represents the ReLU function, W h represents the weight matrix of the hidden layer, [;] represents the concatenation operation, b h Bias vector for the hidden layer.

[0037] As an optional embodiment of the present invention, optionally, introducing a transfer learning mechanism to optimize the behavior prediction model in step S4 includes:

[0038] S401, obtaining a deviation of the behavior prediction model performance based on the difference between the historical prediction data and the current prediction data;

[0039] S402, selecting a transfer learning strategy based on the performance deviation;

[0040] S403, applying the transfer learning strategy to adjust the parameters or structure of the behavior prediction model;

[0041] S404, evaluating the performance of the behavior prediction model after transfer learning, and determining whether the performance of the behavior prediction model is improved based on the evaluation result;

[0042] If yes, save the model parameters after transfer learning;

[0043] If not, return to step S402 to reselect the transfer learning strategy until the optimal transfer learning strategy is found.

[0044] As an optional embodiment of the present invention, optionally, in step S5, iteratively updating the prediction model based on the user feedback information includes:

[0045] S501, collecting user behavior feedback data in actual use, wherein the behavior feedback data includes user satisfaction, number of behavior corrections, and behavior prediction accuracy;

[0046] S502, calculating the performance index of the behavior prediction model based on the behavior feedback data;

[0047] S503, determining whether the performance indicator meets a preset threshold;

[0048] If satisfied, keep the current behavior prediction model parameters unchanged;

[0049] If not, adjust the model parameters or introduce new feature variables according to the behavior feedback data;

[0050] S504, repeating steps S501 to S503 until the performance indicator meets a preset condition.

[0051] In another aspect, the present invention further provides a computer device, comprising:

[0052] processor;

[0053] a memory for storing processor-executable instructions;

[0054] Wherein, the processor is configured to implement a scene-adaptive digital human behavior prediction method when executing the executable instructions.

[0055] In another aspect, the present invention further provides a computer-readable storage medium, comprising:

[0056] a memory having a computer program stored thereon;

[0057] A processor is used to execute the program in the memory to implement a scene-adaptive digital human behavior prediction method.

[0058] Beneficial effects of the present invention: The present invention collects multimodal user interaction data such as text data, voice data and image data, and obtains the feature vectors of these data using a pre-trained multimodal feature extraction model. Based on these feature vectors, the present invention further uses pre-trained scene recognition models and behavior prediction models to predict the behavior of digital humans in different scenarios. In order to improve the prediction accuracy, the present invention introduces a transfer learning mechanism, which optimizes the parameters or structure of the behavior prediction model according to the difference between historical prediction data and current prediction data, thereby improving the prediction accuracy of digital humans in practical applications. In addition, the present invention also collects user feedback information in real time and iteratively updates the prediction model to meet the special needs of different user groups. This scene-adapted digital human behavior prediction method not only improves the prediction accuracy of digital human behavior, but also enables digital humans to better understand and adapt to different scenarios, providing more intelligent and humane services.

[0059] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0061] Figure 1 It is a flow chart of a scene-adaptive digital human behavior prediction method of the present invention. DETAILED DESCRIPTION

[0062] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0063] Example 1

[0064] like Figure 1 As shown, a scene-adaptive digital human behavior prediction method comprises:

[0065] S1. Collecting multimodal user interaction data and preprocessing the multimodal user interaction data; the multimodal user interaction data includes text data, voice data and image data;

[0066] It should be noted that in order to more accurately capture the behavior of digital humans, the present invention will clean, denoise and normalize the multimodal user interaction data in the preprocessing stage. For text data, operations such as word segmentation, stop word removal and stem extraction will be performed; for voice data, voice enhancement, noise suppression and voice endpoint detection will be performed; for image data, image enhancement, face detection and other steps will be performed. These preprocessing steps can significantly improve the efficiency and accuracy of subsequent feature extraction and model training.

[0067] S2. Acquire a multimodal feature vector in the multimodal user interaction data based on a pre-trained multimodal feature extraction model;

[0068] It should be noted that the multimodal feature extraction model of this embodiment includes multiple sub-models, each of which focuses on extracting features from data of one modality. For example, there is a text feature extraction sub-model that specifically processes text data, which can capture key features such as keywords, phrases, and contextual information from text data. Similarly, there is a voice feature extraction sub-model for voice data, which can analyze the characteristics of voice such as pitch, speech speed, and volume, so as to understand the speaker's emotions and intentions. For image data, the image feature extraction sub-model focuses on identifying visual elements such as objects, faces, and actions in the image. These sub-models are trained and optimized through advanced technologies such as deep learning and neural networks to ensure that the feature vectors of their respective modalities can be extracted efficiently and accurately. The features of different modalities extracted by different sub-models are then fused to obtain a multimodal feature vector. This vector contains information from multiple modalities such as text, voice, and image, and can fully reflect the behavior and intention of the user when interacting with the digital human. By inputting the multimodal feature vector into the subsequent scene recognition model and behavior prediction model, the present invention can achieve accurate prediction of the behavior of the digital human in different scenarios.

[0069] S3, obtaining a scene recognition result using a pre-trained scene recognition model based on the multimodal feature vector, and predicting the behavior of the digital human using a behavior prediction model based on the scene recognition result to obtain a digital human behavior prediction result;

[0070] It should be noted that both the scene recognition model and the behavior prediction model are built based on a deep learning framework and are obtained through training with a large amount of historical data. The scene recognition model can identify the specific scenes when the user interacts with the digital person, such as office, home, outdoor, etc., so as to provide contextual information for the behavior prediction model. The behavior prediction model predicts the behavior that the digital person may take in the scene based on the scene recognition results; specific behaviors include but are not limited to greeting, asking questions, answering questions, expressing emotions, providing suggestions or performing specific tasks. In order to improve the accuracy of behavior prediction, this embodiment also considers the influence of time factors and external environment on the behavior of digital people. For example, in a specific time period (such as working hours or rest time) or under specific environmental conditions (such as light brightness, noise level, etc.), the behavior of the digital person will be different. Therefore, the present invention incorporates the consideration of time factors and external environmental factors into the behavior prediction model, so that the prediction results are more in line with the actual situation.

[0071] S4. Based on the digital human behavior prediction results, a transfer learning mechanism is introduced to optimize the behavior prediction model:

[0072] It should be noted that the transfer learning mechanism can automatically adjust the parameters or structure of the behavior prediction model according to the difference between the historical prediction data and the current prediction data, thereby improving the adaptability and prediction accuracy of the model. Specifically, transfer learning can be achieved in the following ways: one is model-based migration, that is, using the existing pre-trained model as a starting point, and adapting to the new prediction task by fine-tuning the model parameters; the second is feature-based migration, that is, extracting effective features from the existing model and using it in the new prediction task; the third is instance-based migration, that is, according to the requirements of the new task, the existing data set is screened and weighted to construct a data set that better meets the requirements of the new task. Through these transfer learning strategies, the present invention can achieve continuous optimization of the behavior prediction model and improve its prediction performance in different scenarios.

[0073] S5. Collect user feedback information in real time, and iteratively update the prediction model based on the user feedback information.

[0074] It should be noted that user feedback information is an important basis for evaluating and improving the performance of the prediction model. In practical applications, feedback information such as user satisfaction, number of behavior corrections, and accuracy of behavior prediction can intuitively reflect the performance of the prediction model in actual scenarios. Therefore, the present invention collects these feedback information in real time and iteratively updates the prediction model to continuously improve its prediction accuracy and adaptability. Specifically, the present invention first collects the user's behavioral feedback data in actual use, which includes but is not limited to user satisfaction scores, number of behavior corrections, and accuracy of behavior prediction. Then, based on these feedback data, the performance indicators of the prediction model, such as accuracy, recall rate, F1 score, etc., are calculated to comprehensively evaluate the performance of the model. Next, the present invention determines whether these performance indicators meet the preset thresholds. If they meet, it means that the performance of the current prediction model is good enough and the current model parameters can be kept unchanged; if they do not meet, it means that there is still room for improvement in the model, and it is necessary to adjust the model parameters or introduce new feature variables according to the feedback data. In the process of adjusting the model parameters or introducing new feature variables, the present invention will make full use of the valuable information in the user feedback information, such as the behavior patterns with low user satisfaction and the behavior types with frequent corrections, to optimize the model in a targeted manner. At the same time, new model structures and algorithms will be constantly tried to find a better prediction model. Finally, the present invention will repeat the above steps, i.e., collect user feedback information in real time, calculate performance indicators, determine whether the performance meets the preset conditions, adjust model parameters or introduce new feature variables, etc., until the performance indicators of the prediction model meet the preset conditions or reach the optimal state. In this way, the present invention can achieve continuous iterative updates of the prediction model to meet the special needs of different user groups and improve the prediction accuracy and intelligence level of digital human behavior.

[0075] In summary, this embodiment proposes an innovative scenario-adaptive digital human behavior prediction method, which not only combines the collection and processing of multimodal user interaction data, but also introduces a pre-trained multimodal feature extraction model, a scenario recognition model, and a behavior prediction model to achieve accurate prediction of digital human behavior in different scenarios. In addition, through the introduction of a transfer learning mechanism and the real-time collection of user feedback information, the present invention can continuously optimize the prediction model and improve its prediction performance and adaptability in different scenarios.

[0076] As an optional embodiment of the present invention, optionally, obtaining the multimodal feature vector in the multimodal user interaction data in step S2 includes:

[0077] S101, extracting semantic features from the text data based on a pre-trained BERT model;

[0078] It should be noted that the BERT model (Bidirectional Encoder Representations from Transformers) is a pre-trained deep bidirectional model that can efficiently capture the semantic features in the text by understanding the contextual information of the text data. In step S101, the text data is processed using the BERT model to extract key semantic information such as keywords, phrases, and logical relationships between sentences, providing strong support for subsequent scene recognition and behavior prediction. At the same time, the powerful generalization ability of the BERT model enables it to adapt to text data of different fields and styles, improving the accuracy and applicability of feature extraction.

[0079] S102, converting the speech data into text based on a speech recognition model, and extracting emotional features;

[0080] It should be noted that in step S102, the voice data is first input into a trained speech recognition model, which can accurately convert the voice signal into the corresponding text content. Subsequently, the converted text is subjected to sentiment analysis using natural language processing technology to extract emotional features such as joy, sadness, anger, etc. These emotional features are of great significance for understanding the user's emotional state and predicting their behavioral responses. In the process of extracting multimodal feature vectors, the present invention also fully considers the correlation and complementarity between different modal data, and organically combines feature vectors from different modalities through an effective feature fusion strategy to form a more comprehensive and accurate multimodal feature representation. This representation method not only improves the efficiency and accuracy of feature extraction, but also provides a more reliable foundation for subsequent scene recognition and behavior prediction.

[0081] S103, extracting visual features from the image data based on a face recognition model;

[0082] It should be noted that in step S103, the image data is processed using advanced face recognition technology, which can accurately capture key visual features such as the outline, expression and movement of the face. These visual features are crucial for identifying the user's identity, emotional state and interaction intention. For example, by analyzing the user's facial expression, it is possible to determine whether the user is happy, surprised or confused, thereby predicting his or her behavioral response. Similarly, by analyzing the user's body movements, such as gestures and postures, the user's intentions and needs can be further understood. In the process of extracting multimodal feature vectors, the present invention uses a variety of advanced deep learning models and algorithms to ensure that useful feature information can be extracted from multimodal data efficiently and accurately. These feature information not only includes the user's behavior and intention when interacting with the digital human, but also reflects the correlation and complementarity between different modal data. By organically integrating and representing these feature information, the present invention can achieve accurate prediction and intelligent services for the behavior of digital humans in different scenarios.

[0083] S104, fusing the semantic features, taking the emotional features and the visual features based on the attention mechanism to obtain a multimodal feature vector.

[0084] It should be noted that the attention mechanism is an effective deep learning technology that can help the model automatically focus on the information that is most important for the current task when processing complex input data. In step S104, the attention mechanism is used to fuse semantic features, emotional features and visual features, and the weights of different features in the final decision can be dynamically adjusted to ensure that the prediction model can more accurately capture the key information when the user interacts with the digital human. Specifically, the attention mechanism assigns a corresponding attention weight to each feature according to its importance. This weight reflects the contribution of the feature to the prediction of the behavior of the digital human. During the fusion process, the model will give priority to those features with higher attention weights and ignore those with lower scores. In this way, the attention mechanism can make the prediction model more focused on the information that is most critical to the current task, thereby improving the accuracy and robustness of the prediction.

[0085] As an optional embodiment of the present invention, optionally, the expression for obtaining the multimodal feature vector in step S104 is:

[0086]

[0087]

[0088] in, represents the mapped semantic feature vector, W T represents the weight matrix of semantic features, T represents semantic features, b TThe bias vector representing the semantic feature, represents the mapped sentiment feature vector, W A Represents the weight matrix of emotional features, A represents emotional features, b A The bias vector representing the sentiment feature, represents the mapped visual feature vector, W V represents the weight matrix of visual features, V represents visual features, b V Represents the bias vector of visual features, α T represents the attention weight of the semantic feature, exp( ) represents the exponential function, and Both represent weight vectors, b′ T , b′ A and b′ V express and The corresponding bias term, F represents the fused multimodal feature vector, α A represents the attention weight of the sentiment feature, α V represents the attention weight of the visual feature, .

[0089] It should be noted that α T +α A +α V =1; Through the above expression, the fused multimodal feature vector can be calculated. This vector not only contains semantic, emotional and visual information, but also effectively integrates and weights these three aspects of information through the attention mechanism. In this way, in the subsequent scene recognition and behavior prediction process, the model can more accurately understand and utilize the multimodal data when the user interacts with the digital human, thereby improving the accuracy and intelligence of the prediction. In addition, this expression also has certain flexibility and scalability, and the weights and bias vectors of different features can be adjusted according to actual needs to meet the needs of different scenarios and tasks.

[0090] As an optional embodiment of the present invention, optionally, obtaining the digital human behavior prediction result in step S3 includes:

[0091] S301, collecting historical behavior data of digital humans, and sorting the historical behavior data according to time series to obtain a time series data set;

[0092] It should be noted that the time series data set can reflect the behavior patterns and rules of digital humans at different time points, and is an important basis for behavior prediction. In step S301, this embodiment constructs a complete and ordered time series data set by collecting the historical behavior data of digital humans and sorting them according to the time series. This data set contains all the behavior records of digital humans in the past period of time, such as movements, expressions, and language, and provides rich data support for subsequent behavior prediction. In the process of collecting historical behavior data, this embodiment fully considers the diversity and integrity of the data, ensuring that the data set can fully reflect the behavioral characteristics and patterns of digital humans. At the same time, by sorting the data in time series, this embodiment further improves the orderliness and predictability of the data, providing a more reliable data basis for subsequent behavior prediction algorithms.

[0093] S302, extracting digital human behavior-related features from the time series data set;

[0094] It should be noted that this embodiment uses advanced deep learning models such as long short-term memory networks (LSTM) to extract the behavior-related features of digital humans in time series data sets. These features include but are not limited to movement frequency, expression changes, language patterns, etc., which can reflect the behavioral laws and characteristics of digital humans in different scenarios. In the process of extracting features, this embodiment makes full use of the powerful learning ability of the deep learning model to effectively process and analyze complex time series data, thereby obtaining a more accurate and comprehensive behavioral feature representation. These feature representations not only provide strong support for subsequent behavior predictions, but also help to better understand the behavioral patterns and laws of digital humans, and provide a basis for their intelligent services.

[0095] S303, obtaining the digital human behavior trend based on the behavior-related characteristics;

[0096] It should be noted that in step S303, by conducting an in-depth analysis of the extracted behavior-related features, this embodiment can predict the behavior trend of the digital human in the future. This prediction process not only takes into account the current behavior state of the digital human, but also combines its historical behavior data, thereby achieving a comprehensive understanding and grasp of the digital human's behavior pattern. Specifically, this embodiment uses advanced prediction algorithms, such as time series analysis, machine learning, etc., to process behavior-related features to predict the possible behavior path and pattern of the digital human. This prediction result is of great significance for identifying potential risks and opportunities in advance and optimizing the digital human behavior strategy.

[0097] S304, based on the behavior trend and scene recognition results, using the trained behavior prediction model to predict the behavior of the digital human, and obtain a prediction probability distribution of the digital human behavior;

[0098] It should be noted that in step S304, this embodiment combines the behavior trend with the scene recognition result as the input of the behavior prediction model. This strategy fully considers the differences and diversity of the behavior patterns of digital humans in different scenes, so that the prediction model can more accurately capture these differences and give behavior prediction results that conform to the actual situation. Specifically, by combining the behavior trend with the scene information, the model can comprehensively consider the current behavior state, historical behavior pattern and specific scene of the digital human, so as to achieve accurate prediction of the future behavior of the digital human. This prediction result is given in the form of probability distribution, reflecting the different behaviors that the digital human may take and their corresponding probabilities. In this way, in the subsequent service and interaction process, the system can make corresponding preparations and adjustments in advance according to the prediction results to provide a more intelligent and personalized service experience. At the same time, the prediction results can also provide an important reference for the optimization and adjustment of the digital human behavior strategy, helping the system to better understand and adapt to the needs and preferences of users.

[0099] S305: Based on the digital human behavior prediction probability distribution, select the behavior with the highest probability as the digital human behavior prediction result.

[0100] It should be noted that in step S305, this embodiment adopts a simple and effective strategy, that is, selecting the behavior with the highest probability as the prediction result of the digital human behavior. This strategy is based on the basic principle of probability theory, which believes that under given conditions, the event with the highest probability of occurrence is most likely to become a reality. Therefore, after obtaining the probability distribution of the digital human behavior prediction, this embodiment compares the prediction probabilities of different behaviors and selects the behavior with the highest probability as the final prediction result. This result not only reflects the most likely behavior of the digital human in the current scenario, but also provides clear guidance for subsequent services and interactions. For example, in the intelligent customer service scenario, if the prediction result shows that the digital human is most likely to answer the user's question, the system can prepare the corresponding answer content in advance to improve the efficiency of the interaction and user experience. Similarly, in the intelligent navigation scenario, if the prediction result shows that the digital human is most likely to choose a certain path, the system can plan the navigation route in advance to provide more intelligent services.

[0101] As an optional embodiment of the present invention, optionally, the expression for obtaining the behavior trend of the digital human based on the behavior-related features is:

[0102] D t =η·M t +β·S t +γ·E t

[0103] M t =λ·(Yt -Y t-1 )+(1-λ)·M t-1

[0104]

[0105]

[0106] Among them, D t represents the behavior trend, η, β and γ represent the average weight coefficients, M t represents the momentum of the behavior change at time t, S t Indicates seasonal changes in behavior, E t represents the impact of external factors on behavior, λ represents the momentum smoothing factor, and Y t represents the behavior label of the digital human at time t, Y t-1 The digital human behavior label at time t-1, M t-1 The momentum of the behavior change at time t-1, K represents the length of the seasonal cycle, φ k represents the seasonal weight coefficient, Y t-k·P represents the behavior label at time tk·P, P represents the basic unit of seasonal cycle, N represents the number of external influencing factors, θ i represents the weight coefficient of the i-th external influencing factor, E i,t Represents the value of the i-th external influencing factor at time t.

[0107] It should be noted that this expression comprehensively considers multiple aspects of digital human behavior, including the momentum of behavioral changes, seasonal changes, and the impact of external factors. By introducing the average weight coefficients η, β, and γ, this expression can balance the contribution of different factors to behavioral trends, thereby more accurately reflecting the behavioral patterns of digital humans. The momentum term captures the changing trend of behavior over time, which helps to identify the continuity and direction of change of behavior. The seasonal change term takes into account the cyclical laws that may exist in behavior at different time points, such as changes in the morning and evening of the day, and the difference between weekdays and weekends in the week. The external factor term represents external events or conditions that affect the behavior of digital humans, such as weather, holidays, special events, etc.

[0108] As an optional embodiment of the present invention, optionally, in step S04, the expression for obtaining the prediction probability distribution of digital human behavior is:

[0109] P t =f(D t ,H t )

[0110] P t =softmax(W o ·h t+b o )

[0111] h t =σ(W h ·[D t ;H t ]+b h )

[0112] Among them, P t represents the probability distribution of various behaviors that the digital human may take at time t, f( ) represents the behavior prediction model, and D t Indicates behavioral trend, H t represents the scene recognition result at time t, softmax() represents the activation function, W o represents the weight matrix of the output layer, h t represents the hidden layer output at time t, b o represents the bias vector of the output layer, σ( ) represents the ReLU function, W h represents the weight matrix of the hidden layer, [;] represents the concatenation operation, b h Bias vector for the hidden layer.

[0113] It should be noted that this expression is the core component of the behavior prediction model, which defines how to calculate the probability distribution of various behaviors that the digital human may take from the input behavior trends and scene recognition results. In this expression, the behavior prediction model receives the behavior trends and scene recognition results as input, and finally outputs a probability distribution through a series of calculations and transformations. This probability distribution reflects the different behaviors taken by the digital human in a given scenario and their corresponding probabilities. Parameters such as the activation function, the weight matrix of the output layer, and the bias vector jointly determine the output and prediction performance of the model. By adjusting these parameters, the prediction accuracy and generalization ability of the model can be optimized to better adapt to the needs of different scenarios and tasks. At the same time, through multiple layers of nonlinear transformations, the model can extract useful features and information from the original input, and make accurate predictions and decisions based on these features and information.

[0114] As an optional embodiment of the present invention, optionally, introducing a transfer learning mechanism to optimize the behavior prediction model in step S4 includes:

[0115] S401, obtaining a deviation of the behavior prediction model performance based on the difference between the historical prediction data and the current prediction data;

[0116] It should be noted that in step S401, the transfer learning mechanism is introduced to optimize the performance of the behavior prediction model. Specifically, this embodiment first analyzes the difference between the historical prediction data and the current prediction data. This difference may be due to a variety of factors, such as changes in the behavior patterns of digital humans, increased scene complexity, or changes in the external environment. By conducting an in-depth analysis of these differences, this embodiment can identify deviations in the performance of the behavior prediction model. This step is the basis for the application of the transfer learning mechanism, which helps to determine the direction and focus of model optimization.

[0117] S402, selecting a transfer learning strategy based on the performance deviation;

[0118] It should be noted that in step S402, based on the identified performance deviation, this embodiment selects a suitable transfer learning strategy to optimize the behavior prediction model. The choice of transfer learning strategy depends on the specific type and degree of the deviation, as well as the similarity and correlation between the current task and the previous task. For example, if the deviation is mainly due to changes in the behavior pattern of the digital human, a strategy based on feature migration can be selected to migrate useful features learned in the previous task to the current task to improve the adaptability and accuracy of the model. If the deviation is caused by the increase in scene complexity, a strategy based on model migration can be selected, using the model trained in the previous task as a starting point, and adapting to new complex scenes through fine-tuning or expansion. In addition, multiple transfer learning strategies can be combined to achieve more comprehensive model optimization.

[0119] S403, applying the transfer learning strategy to adjust the parameters or structure of the behavior prediction model;

[0120] It should be noted that in step S403, the transfer learning strategy is specifically applied to the optimization process of the behavior prediction model. This step aims to correct performance deviations and improve the prediction accuracy of the model by adjusting the parameters or structure of the model. Specifically, according to the selected transfer learning strategy, the behavior prediction model is adjusted accordingly in this embodiment. These adjustments include modifying the parameters such as the weight matrix and bias vector of the model, or changing the structure of the model, such as increasing or decreasing the number of layers, changing the type of layer or the connection method, etc. Through these adjustments, the model can better adapt to the needs of the current task and reduce the performance degradation caused by the difference between historical data and current data. In addition, the application of the transfer learning strategy also helps to improve the generalization ability of the model, enabling it to better handle unseen scenes and data, thereby further improving the performance and reliability of digital human behavior prediction.

[0121] S404, evaluating the performance of the behavior prediction model after transfer learning, and determining whether the performance of the behavior prediction model is improved based on the evaluation result;

[0122] If yes, save the model parameters after transfer learning;

[0123] If not, return to step S402 to reselect the transfer learning strategy until the optimal transfer learning strategy is found.

[0124] It should be noted that in step S404, the performance of the behavior prediction model after transfer learning is comprehensively evaluated. This evaluation process is intended to verify the effectiveness of the transfer learning strategy and determine whether the model performance is improved. The evaluation indicators include prediction accuracy, generalization ability, computational efficiency and other aspects to ensure that the model can meet actual needs in multiple dimensions. If the evaluation results show that the model performance has improved, the model parameters after transfer learning will be saved for subsequent use. These parameters contain the optimization results of the model in the transfer learning process, which can ensure that the model performs better on new tasks and data. However, if the evaluation results show that the model performance has not improved, or the improvement does not meet expectations, then this embodiment will return to step S402 and reselect the transfer learning strategy. The iterative process of this step will continue until the optimal transfer learning strategy is found. In this way, the present embodiment can continuously optimize the performance of the behavior prediction model to ensure that it can adapt to the ever-changing tasks and data requirements. In addition, throughout the entire process of transfer learning, the present embodiment also focuses on maintaining the stability and reliability of the model. When adjusting model parameters or structure, the robustness and generalization ability of the model are fully considered to avoid performance degradation due to overfitting or underfitting.

[0125] As an optional embodiment of the present invention, optionally, in step S5, iteratively updating the prediction model based on the user feedback information includes:

[0126] S501, collecting user behavior feedback data in actual use, wherein the behavior feedback data includes user satisfaction, number of behavior corrections, and behavior prediction accuracy;

[0127] It should be noted that in step S501, the user's behavior feedback data in actual use is collected and used to iteratively update the prediction model. These behavior feedback data cover multiple aspects, including user satisfaction, behavior correction times and behavior prediction accuracy. User satisfaction reflects the user's acceptance and satisfaction with the model prediction results, and is one of the important indicators for evaluating model performance. The number of behavior corrections records the number of times the user corrects after the model prediction error, which helps to identify the blind spots and deficiencies of the model in the prediction. The behavior prediction accuracy directly measures the consistency between the model prediction results and the actual behavior, and is a key indicator for evaluating the model prediction performance. By collecting these behavior feedback data, this embodiment can comprehensively analyze and evaluate the prediction model and identify the problems and deficiencies of the model in the prediction. These data provide strong support for subsequent model iteration and update, which helps to continuously optimize the performance of the model and improve the accuracy of the prediction. After collecting enough behavior feedback data, this embodiment will iteratively update the prediction model based on these data. This process includes adjusting the parameters of the model, optimizing the structure of the model or introducing new features. Through continuous iterative updates, the model can gradually adapt to the user's behavior patterns and demand changes, and improve the accuracy and reliability of the prediction. At the same time, this can also enhance the generalization ability of the model, enabling it to better handle unseen scenarios and data, thereby further improving the performance of digital human behavior prediction and user experience.

[0128] S502, calculating the performance index of the behavior prediction model based on the behavior feedback data;

[0129] It should be noted that in step S502, based on the collected behavior feedback data, this embodiment calculates multiple performance indicators of the behavior prediction model. These performance indicators are intended to comprehensively evaluate the performance of the model and identify the advantages and disadvantages of the model in the prediction. Specifically, these performance indicators include but are not limited to prediction accuracy, user satisfaction improvement rate, behavior correction rate reduction, etc. Prediction accuracy is a key indicator to measure the consistency between the model prediction results and the actual behavior, which directly reflects the prediction ability of the model. The user satisfaction improvement rate evaluates the impact of model optimization on user acceptance by comparing the user satisfaction data before and after the model update. The behavior correction rate reduction records whether the number of times the model is corrected by the user due to prediction errors after the iterative update is reduced, which helps to verify the effect of model optimization in reducing prediction errors. By calculating these performance indicators, this embodiment can quantitatively evaluate the performance of the prediction model and provide strong data support for subsequent model optimization and iterative updates. At the same time, these performance indicators can also be used as an intuitive reflection of the improvement of model performance, showing users the improvement effect and practical application value of the model in the optimization process.

[0130] S503, determining whether the performance indicator meets a preset threshold;

[0131] If satisfied, keep the current behavior prediction model parameters unchanged;

[0132] If not, adjust the model parameters or introduce new feature variables according to the behavior feedback data;

[0133] It should be noted that in step S503, the calculated behavior prediction model performance index is strictly judged. This judgment process is intended to determine whether the model performance has reached the preset standards and requirements. The preset threshold is determined based on factors such as actual application scenarios and requirements, as well as historical data of model performance. These thresholds reflect the expected level of the model in terms of prediction accuracy, user satisfaction, and behavior correction rate. If the performance index meets the preset threshold, it means that the performance of the current behavior prediction model has reached the expected requirements. In this case, in order to maintain the stability and reliability of the model, this embodiment will keep the parameters of the current behavior prediction model unchanged. This means that the model will continue to predict in the current state without additional adjustment or optimization. However, if the performance index does not meet the preset threshold, it means that the current behavior prediction model has deficiencies in terms of prediction accuracy, user satisfaction, or behavior correction rate. In this case, this embodiment will adjust the model parameters or introduce new feature variables based on the behavior feedback data. This step is intended to improve the prediction performance and user satisfaction of the model by optimizing the model structure and parameters. Adjusting the model parameters includes modifying the weight matrix, bias vector, etc., while introducing new feature variables involves adding new input features or improving feature extraction methods, etc. Through these adjustments and optimizations, the model can better adapt to user behavior patterns and changes in demand, and improve the accuracy and reliability of predictions.

[0134] S504, repeating steps S501 to S503 until the performance indicator meets a preset condition.

[0135] It should be noted that, in step S504, in order to ensure that the performance of the behavior prediction model can be continuously improved and meet the needs of practical applications, this embodiment adopts an iterative update process. This process forms a closed-loop feedback mechanism by repeating steps S501 to S503. In each iteration, the user's behavior feedback data in actual use is collected, and the performance indicators of the behavior prediction model are calculated based on these data. Then, these performance indicators will be strictly judged to determine whether the model performance has reached the preset standards and requirements. If the performance indicator does not meet the preset threshold, the model parameters will be adjusted or new feature variables will be introduced according to the behavior feedback data to optimize the model structure and improve the prediction performance. This process will continue until the performance indicator meets the preset conditions. In this way, this embodiment can continuously iteratively update and optimize the behavior prediction model so that it can better adapt to the user's behavior pattern and demand changes. At the same time, this can also ensure that the model can show good performance in multiple dimensions, including prediction accuracy, user satisfaction, generalization ability, etc. Ultimately, this will help improve the performance and user experience of digital human behavior prediction and provide more intelligent and reliable support for practical application scenarios.

[0136] Example 2

[0137] A computer device comprising:

[0138] processor;

[0139] a memory for storing processor-executable instructions;

[0140] Wherein, the processor is configured to implement a scene-adaptive digital human behavior prediction method when executing the executable instructions.

[0141] It should be noted that the computer device includes: a processor, a memory, and the computer device may also include one or more of a multimedia component, an input / output (I / O) interface, and a communication component.

[0142] The processor is used to control the overall operation of the computer device to complete all or part of the steps in the above-mentioned scene-adapted digital human behavior prediction method.

[0143] The memory is used to store various types of data to support operations on the computer device, which data may include, for example, instructions for any application or method operating on the computer device, as well as application-related data; the memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0144] The multimedia component may include a screen and an audio component, wherein the screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals; for example, the audio component may include a microphone for receiving external audio signals, and the received audio signals may be further stored in a memory or sent via a communication component; the audio component also includes at least one speaker for outputting audio signals.

[0145] The I / O interface provides an interface between the processor and other interface modules, which may be a keyboard, a mouse, buttons, etc.; these buttons may be virtual buttons or physical buttons.

[0146] The communication component is used for wired or wireless communication between the computer device and other devices; wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G or 5G, or a combination of one or more of them, so the corresponding communication component may include: Wi-Fi module, Bluetooth module, NFC module, mobile phone communication module.

[0147] As a preferred solution of this embodiment, the computer device can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned scene-adapted digital human behavior prediction method.

[0148] Example 3

[0149] A computer-readable storage medium, comprising:

[0150] a memory having a computer program stored thereon;

[0151] A processor is used to execute the program in the memory to implement a scene-adaptive digital human behavior prediction method.

[0152] It should be noted that the electronic device according to the embodiment of the present disclosure includes a processor and a memory for storing instructions executable by the processor, wherein the processor is configured to implement any of the above-mentioned scene-adapted digital human behavior prediction methods when executing the executable instructions.

[0153] Here, it should be noted that the number of processors can be one or more. At the same time, the electronic device of the embodiment of the present disclosure may also include an input device and an output device. Among them, the processor, memory, input device and output device may be connected through a bus or in other ways, which are not specifically limited here.

[0154] The memory, as a computer-readable storage medium, can be used to store software programs, computer executable programs and various modules, such as the program or module corresponding to the scene-adapted digital human behavior prediction method of the embodiment of the present disclosure. The processor executes various functional applications and data processing of the electronic device by running the software programs or modules stored in the memory.

[0155] The input device can be used to receive input numbers or signals. The signal can be a key signal related to user settings and function control of the device / terminal / server. The output device can include a display device such as a display screen.

[0156] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A scene-adaptive digital human behavior prediction method, characterized in that: The method comprises: S1. Collecting multimodal user interaction data and preprocessing the multimodal user interaction data; the multimodal user interaction data includes text data, voice data and image data; S2. Acquire a multimodal feature vector in the multimodal user interaction data based on a pre-trained multimodal feature extraction model; S3, obtaining a scene recognition result using a pre-trained scene recognition model based on the multimodal feature vector, and predicting the behavior of the digital human using a behavior prediction model based on the scene recognition result to obtain a digital human behavior prediction result; S4. Based on the digital human behavior prediction results, a transfer learning mechanism is introduced to optimize the behavior prediction model: S5. Collect user feedback information in real time, and iteratively update the prediction model based on the user feedback information.

2. The scene-adaptive digital human behavior prediction method according to claim 1, characterized in that: Acquiring the multimodal feature vector in the multimodal user interaction data in step S2 includes: S101, extracting semantic features from the text data based on a pre-trained BERT model; S102, converting the speech data into text based on a speech recognition model, and extracting emotional features; S103, extracting visual features from the image data based on a face recognition model; S104, fusing the semantic features, taking the emotional features and the visual features based on the attention mechanism to obtain a multimodal feature vector.

3. The scene-adaptive digital human behavior prediction method according to claim 2, characterized in that: The expression for obtaining the multimodal feature vector in step S104 is: in, represents the semantic feature vector after mapping, W T represents the weight matrix of semantic features, T represents semantic features, b T The bias vector representing the semantic feature, represents the mapped sentiment feature vector, W A Represents the weight matrix of emotional features, A represents emotional features, b A The bias vector representing the sentiment feature, represents the mapped visual feature vector, W V represents the weight matrix of visual features, V represents visual features, b V Represents the bias vector of visual features, α T represents the attention weight of the semantic feature, exp( ) represents the exponential function, and Both represent weight vectors, α A represents the attention weight of the sentiment feature, α V represents the attention weight of the visual feature, b′ T , b′ A and b′ V express and The corresponding bias term, F represents the fused multimodal feature vector.

4. The scene-adaptive digital human behavior prediction method according to claim 1, characterized in that: The digital human behavior prediction results obtained in step S3 include: S301, collecting historical behavior data of digital humans, and sorting the historical behavior data according to time series to obtain a time series data set; S302, extracting digital human behavior-related features from the time series data set; S303, obtaining the digital human behavior trend based on the behavior-related characteristics; S304, based on the behavior trend and scene recognition results, using the trained behavior prediction model to predict the behavior of the digital human, and obtain a prediction probability distribution of the digital human behavior; S305: Based on the digital human behavior prediction probability distribution, select the behavior with the highest probability as the digital human behavior prediction result.

5. The scene-adaptive digital human behavior prediction method according to claim 4, characterized in that: The expression for obtaining the behavior trend of the digital human based on the behavior-related characteristics is: D t =η·M t +β·S t +γ·E t M t =λ·(Y t -Y t-1 )+(1-λ)·M t-1 Among them, D t represents the behavior trend, η, β and γ represent the average weight coefficients, M t represents the momentum of the behavior change at time t, S t Indicates seasonal changes in behavior, E t represents the impact of external factors on behavior, λ represents the momentum smoothing factor, and Y t represents the behavior label of the digital human at time t, Y t-1 The digital human behavior label at time t-1, M t-1 The momentum of the behavior change at time t-1, K represents the length of the seasonal cycle, φ k represents the seasonal weight coefficient, Y t-k·P represents the behavior label at time tk·P, P represents the basic unit of seasonal cycle, N represents the number of external influencing factors, θ i represents the weight coefficient of the i-th external influencing factor, E i,t Represents the value of the i-th external influencing factor at time t.

6. A scene-adaptive digital human behavior prediction method as claimed in claim 4, characterized in that: The expression for the predicted probability distribution of digital human behavior obtained in step S04 is: P t =f(D t ,H t ) <h2 style=";text-align:left;direction:ltr">P<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> =softmax9W<h2 style=";text-align:left;direction:ltr"> o <h2 style=";text-align:left;direction:ltr"> h<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> +b<h2 style=";text-align:left;direction:ltr"> o <h2 style=";text-align:left;direction:ltr"> ) h t =σ(W h ·[D t ;H t ]+b h ) Among them, P t represents the probability distribution of various behaviors that the digital human may take at time t, f() represents the behavior prediction model, and D t Indicates behavioral trend, H t represents the scene recognition result at time t, softmax() represents the activation function, W o represents the weight matrix of the output layer, h t represents the hidden layer output at time t, b o represents the bias vector of the output layer, σ() represents the ReLU function, and W h represents the weight matrix of the hidden layer, [;] represents the concatenation operation, b h Bias vector for the hidden layer.

7. The scene-adaptive digital human behavior prediction method according to claim 1, characterized in that: Introducing a transfer learning mechanism to optimize the behavior prediction model in step S4 includes: S401, obtaining a deviation of the behavior prediction model performance based on the difference between the historical prediction data and the current prediction data; S402, selecting a transfer learning strategy based on the performance deviation; S403, applying the transfer learning strategy to adjust the parameters or structure of the behavior prediction model; S404, evaluating the performance of the behavior prediction model after transfer learning, and determining whether the performance of the behavior prediction model is improved based on the evaluation result; If yes, save the model parameters after transfer learning; If not, return to step S402 to reselect the transfer learning strategy until the optimal transfer learning strategy is found.

8. The scene-adaptive digital human behavior prediction method according to claim 1, characterized in that: In step S5, iteratively updating the prediction model based on the user feedback information includes: S501, collecting user behavior feedback data in actual use, wherein the behavior feedback data includes user satisfaction, number of behavior corrections, and behavior prediction accuracy; S502, calculating the performance index of the behavior prediction model based on the behavior feedback data; S503, determining whether the performance indicator meets a preset threshold; If satisfied, keep the current behavior prediction model parameters unchanged; If not, adjust the model parameters or introduce new feature variables according to the behavior feedback data; S504, repeating steps S501 to S503 until the performance indicator meets a preset condition.

9. A computer device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement a scene-adapted digital human behavior prediction method according to any one of claims 1 to 8 when executing the executable instructions.

10. A computer-readable storage medium, characterized in that: include: a memory having a computer program stored thereon; A processor is used to execute the program in the memory to implement a scene-adapted digital human behavior prediction method according to any one of claims 1 to 8.

Images