Behavior prediction method, system, device and storage medium based on AI digital human

Abstract

This application relates to a behavior prediction method, system, device, and storage medium based on AI digital humans. The method includes acquiring raw visual data, voice data, and environmental sensor data, and performing preliminary preprocessing to obtain a processed data set; extracting multimodal features from the processed data set to obtain visual features, voice features, and environmental perception features, and performing weighted fusion of each modal feature based on an attention mechanism to generate a multimodal feature representation; based on the multimodal feature representation, using a deep learning model to optimize occlusion compensation, identify and repair occluded areas in the raw visual data, and then generate an optimized feature set; and performing behavior prediction on the optimized feature set using a time series analysis model to generate behavior output related to the user's current interaction state. This application has the effect of improving the efficiency of behavior prediction.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a behavior prediction method, system, device and storage medium based on AI digital human. Background Art

[0002] At present, with the rapid development of artificial intelligence technology, AI digital humans have been widely used in virtual reality, robot interaction, intelligent customer service and other fields. AI digital humans are virtual humans constructed through technologies such as computer vision, and have multiple interactive capabilities such as visual perception and voice recognition. In the task of behavior prediction, AI digital humans usually predict user behavior and reactions by processing and analyzing image data.

[0003] Existing behavior prediction methods mostly rely on single-modal data for modeling and processing, which can complete simple prediction tasks to a certain extent. However, due to the lack of full utilization of different modalities and the correlation and interactivity between modalities, the prediction accuracy is limited, especially in complex interaction scenarios, and cannot adapt to the dynamic changes of multimodal information.

[0004] The above-mentioned existing technical solutions have the following defects: the existing behavior prediction methods fail to fully consider the interactive relationship and dynamic changes between modalities, resulting in low behavior prediction efficiency, so there is room for improvement. Summary of the Invention

[0005] In order to improve the efficiency of behavior prediction, the present application provides a behavior prediction method, system, device and storage medium based on AI digital human.

[0006] The above-mentioned invention objective of this application is achieved through the following technical solutions:

[0007] A behavior prediction method based on AI digital human, the method comprising:

[0008] Obtain raw visual data, voice data, and environmental sensor data, and perform preliminary preprocessing to obtain a processed data set;

[0009] Extracting multimodal features from the processed data set to obtain visual features, speech features, and environmental perception features, and performing weighted fusion of the features of each modality based on an attention mechanism to generate a multimodal feature representation;

[0010] Based on the multimodal feature representation, a deep learning model is used to perform occlusion compensation optimization, identify and repair occluded areas in the original visual data, and then generate an optimized feature set;

[0011] The optimized feature set is subjected to behavior prediction through a time series analysis model to generate behavior output related to the user's current interaction state.

[0012] By adopting the above technical solutions, by acquiring raw visual data, voice data and environmental sensor data and performing preliminary preprocessing, it is possible to integrate multiple information sources, improve the quality and integrity of the data, and thus provide richer and more accurate input data for subsequent feature extraction and analysis; by extracting multimodal features from the processed data set, visual features, voice features and environmental perception features are obtained, and weighted fusion of each modal feature is performed based on the attention mechanism, which can fully consider the importance of different modalities in behavior prediction and improve the accuracy and reliability of the prediction results; by using a deep learning model based on multimodal feature representation to perform occlusion compensation optimization, identifying and repairing occluded areas in the raw visual data, it is possible to overcome the impact of occlusion problems in the data and ensure that the model can operate normally in complex environments, thereby improving the comprehensiveness and accuracy of the prediction; by using a time series analysis model to perform behavior prediction on the optimized feature set, generating behavior output related to the user's current interaction state, it is possible to dynamically adjust behavior predictions, improve the real-time and personalization of system responses, and thus optimize the user interaction experience.

[0013] In one example, the present application may be further configured as follows: performing weighted fusion of features of each modality based on the attention mechanism specifically includes:

[0014] Designing an independent weight matrix for each modal feature, wherein the weight matrix is ​​designed based on the contribution of each modal feature to the behavior prediction task, the contribution is evaluated by a deep learning model during training, and initial weight values ​​are assigned to each modal feature based on the evaluation results;

[0015] Based on the relevance and interactivity of the modal features, a cross-modal attention mechanism is introduced to establish an interactive relationship between the modal features, and the weight matrix of the modal features is adjusted according to the interactive relationship;

[0016] Based on the adjusted weight matrix, the information within each modal feature is weightedly fused through the self-attention mechanism.

[0017] By adopting the above technical solution, by designing an independent weight matrix for each modal feature and designing it according to the contribution of the modal feature in the behavior prediction task, it is possible to accurately evaluate the impact of each modality on the final prediction result, avoid the interference of invalid features, and thus enhance the accuracy and reliability of the prediction model; by introducing the cross-modal attention mechanism, it is possible to capture the interaction between different modalities, improve the depth and accuracy of feature fusion, and thus optimize the comprehensiveness and accuracy of the prediction results.

[0018] In one example, the present application may be further configured as follows: based on the correlation and interactivity of the modal features, a cross-modal attention mechanism is introduced to establish an interactive relationship between the modal features, and the weight matrix of the modal features is adjusted according to the interactive relationship, specifically including:

[0019] Based on the relevance and interactivity of the modal features, a cross-modal attention mechanism is constructed. By calculating the correlation between the modal features, appropriate attention weights are assigned to the modal features, thereby capturing the mutual influence relationship between different modalities.

[0020] Based on the cross-modal attention mechanism, a multi-level attention module is used to process modal features layer by layer, capturing local and global modal interaction information at different levels;

[0021] Based on the modal interaction information, a cross-modal adaptive weighting mechanism is introduced to dynamically adjust the weight matrix according to the contribution of each modal feature.

[0022] By adopting the above technical solution and constructing a cross-modal attention mechanism based on the correlation and interactivity of each modal feature, it is possible to more finely explore the mutual influence relationship between different modalities, thereby improving the model's ability to integrate multimodal data; by processing modal features layer by layer through multi-level attention modules, capturing local and global modal interaction information at different levels, it is possible to deeply understand the complex relationship between modalities, thereby improving the model's ability to predict user behavior; by introducing an adaptive weighting mechanism, the weight matrix is ​​dynamically adjusted according to the contribution of each modal feature, which can flexibly adapt to changes in the importance of features in different scenarios, thereby further improving the accuracy and flexibility of prediction.

[0023] In one example, the present application may be further configured as follows: performing occlusion compensation optimization based on the multimodal feature representation using a deep learning model to identify and repair occluded areas in the original visual data, specifically including:

[0024] Based on the visual features in the multimodal feature representation, locating the occluded area in the visual data through a deep convolutional neural network, and using contextual information and neighborhood features of the occluded area to identify and demarcate the occluded area;

[0025] Combining the speech features and the environmental perception features, a cross-modal deep learning model is used to perform occlusion repair, and a compensation strategy for the occluded area is optimized through an adaptive learning mechanism;

[0026] Utilize recurrent neural networks to predict and optimize occlusion repair in time series. Based on the temporal characteristics of occlusion events, the system performs real-time repair of rapidly changing occluded areas in dynamic environments, and optimizes the restoration effect of occluded areas based on the continuity of previous and next frames.

[0027] The repaired occluded area is refined based on a generative adversarial network, and natural visual content is generated through a generator.

[0028] By adopting the above technical solution, by using a deep convolutional neural network based on the visual features in the multimodal feature representation to locate the occluded areas in the visual data, the occluded areas can be identified and calibrated efficiently and accurately, avoiding the occlusion problem from affecting subsequent analysis; by combining speech features and environmental perception features, a cross-modal deep learning model is used for occlusion repair, which can use multiple modal information to compensate for occlusion and improve the naturalness and accuracy of the repair effect; by using a recurrent neural network to predict and optimize occlusion repair in time series, the changing occluded areas in a dynamic environment can be repaired in real time, thereby improving the system's adaptability and real-time performance to rapidly changing scenes; by refining the repaired occluded areas through a generative adversarial network, more natural content that conforms to visual features can be generated, thereby improving the realism and visual quality of the repaired areas.

[0029] In one example, the present application may be further configured as follows: locating the occluded area in the visual data by using a deep convolutional neural network, and identifying and calibrating the occluded area by using the contextual information and neighborhood features of the occluded area, specifically including:

[0030] By combining the context information in the image with a multi-scale convolution operation, the surrounding environment features of the occluded area are captured;

[0031] Based on the neighborhood features of the occluded area, the spatial relationship between adjacent pixels is extracted, and then the boundary of the occluded area is calibrated to determine the position and range of the occluded area.

[0032] By adopting the above technical solution, through multi-scale convolution operations and combining with the contextual information in the image, the environmental features around the occluded area can be effectively captured, thereby providing more comprehensive occluded area information and helping to improve the restoration quality; by extracting the spatial relationship between adjacent pixels based on the neighborhood features of the occluded area, the boundary and range of the occluded area can be accurately calibrated, the erroneous restoration area can be reduced, and the restoration accuracy and reliability can be improved.

[0033] In one example, the present application may be further configured as follows: performing behavior prediction on the optimized feature set using a time series analysis model to generate a behavior output related to the user's current interaction state specifically includes:

[0034] Based on the optimized feature set, the data in the time dimension is encoded to capture the pattern of interaction behavior evolving over time, thereby obtaining time series features;

[0035] A time series analysis model is used to model the time series features, capture the impact of historical behavior on future behavior, and predict user behavior at the current moment and / or future moments;

[0036] Based on the prediction results of the time series analysis model, a behavior output related to the user's current interaction state is generated.

[0037] By adopting the above technical solutions and encoding data in the time dimension, we can capture the patterns of interaction behavior evolving over time, thereby understanding the temporal laws of user behavior; by adopting a time series analysis model to model time series features, we can better capture the impact of historical behavior on future behavior, thereby improving the accuracy of behavior prediction; by generating behavioral outputs related to the user's current interaction status based on the prediction results of the time series analysis model, we can adjust the system response in a timely manner, optimize the user experience, and enhance the intelligence and personalization of the interaction.

[0038] The second object of the present invention is achieved through the following technical solutions:

[0039] A behavior prediction system based on AI digital humans, comprising:

[0040] The data acquisition and processing module is used to obtain raw visual data, voice data, and environmental sensor data, and perform preliminary preprocessing to obtain a processed data set;

[0041] A feature extraction module is used to extract multimodal features from the processed data set to obtain visual features, speech features, and environmental perception features, and to perform weighted fusion of the features of each modality based on an attention mechanism to generate a multimodal feature representation;

[0042] an occlusion compensation optimization module, configured to perform occlusion compensation optimization based on the multimodal feature representation using a deep learning model, identify and repair occluded areas in the original visual data, and generate an optimized feature set;

[0043] The behavior prediction module is used to perform behavior prediction on the optimized feature set through a time series analysis model and generate behavior output related to the user's current interaction state.

[0044] By adopting the above technical solutions, by acquiring raw visual data, voice data and environmental sensor data and performing preliminary preprocessing, it is possible to integrate multiple information sources, improve the quality and integrity of the data, and thus provide richer and more accurate input data for subsequent feature extraction and analysis; by extracting multimodal features from the processed data set, visual features, voice features and environmental perception features are obtained, and weighted fusion of each modal feature is performed based on the attention mechanism, which can fully consider the importance of different modalities in behavior prediction and improve the accuracy and reliability of the prediction results; by using a deep learning model based on multimodal feature representation to perform occlusion compensation optimization, identifying and repairing occluded areas in the raw visual data, it is possible to overcome the impact of occlusion problems in the data and ensure that the model can operate normally in complex environments, thereby improving the comprehensiveness and accuracy of the prediction; by using a time series analysis model to perform behavior prediction on the optimized feature set, generating behavior output related to the user's current interaction state, it is possible to dynamically adjust behavior predictions, improve the real-time and personalization of system responses, and thus optimize the user interaction experience.

[0045] The third objective of this application is achieved through the following technical solutions:

[0046] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for predicting behavior based on AI digital humans are implemented.

[0047] The fourth objective of this application is achieved through the following technical solutions:

[0048] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned behavior prediction method based on AI digital humans.

[0049] In summary, this application has the following beneficial technical effects:

[0050] 1. By acquiring raw visual data, voice data, and environmental sensor data and performing preliminary preprocessing, it is possible to integrate multiple information sources, improve the quality and integrity of the data, and thus provide richer and more accurate input data for subsequent feature extraction and analysis; by extracting multimodal features from the processed data set, visual features, voice features, and environmental perception features are obtained, and weighted fusion of each modality feature is performed based on the attention mechanism, which can fully consider the importance of different modalities in behavior prediction and improve the accuracy and reliability of the prediction results; by using a deep learning model based on multimodal feature representation to optimize occlusion compensation, identify and repair occluded areas in the raw visual data, it is possible to overcome the impact of occlusion problems in the data and ensure that the model can operate normally in complex environments, thereby improving the comprehensiveness and accuracy of the prediction; through the time series analysis model, the optimized feature set is used to predict behavior, generate behavior output related to the user's current interaction state, and dynamically adjust the behavior prediction to improve the real-time and personalization of the system response, thereby optimizing the user interaction experience;

[0051] 2. By designing independent weight matrices for each modal feature and basing the weighting on the modal feature's contribution to the behavior prediction task, we can accurately assess the impact of each modality on the final prediction result, avoid interference from invalid features, and thus enhance the accuracy and reliability of the prediction model. By introducing a cross-modal attention mechanism, we can capture the interactions between different modalities, improve the depth and accuracy of feature fusion, and thus optimize the comprehensiveness and accuracy of the prediction results.

[0052] 3. By constructing a cross-modal attention mechanism based on the correlation and interactivity of each modal feature, the mutual influence relationship between different modalities can be more finely explored, thereby improving the model's ability to integrate multimodal data; by processing modal features layer by layer through multi-level attention modules, local and global modal interaction information can be captured at different levels, which can deeply understand the complex relationship between modalities, thereby improving the model's ability to predict user behavior; by introducing an adaptive weighting mechanism, the weight matrix is ​​dynamically adjusted according to the contribution of each modal feature, which can flexibly adapt to the changes in the importance of features in different scenarios, thereby further improving the accuracy and flexibility of predictions. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of a behavior prediction method based on AI digital human in one embodiment of the present application;

[0054] Figure 2 This is a flowchart for implementing step S20 in the behavior prediction method based on AI digital human in one embodiment of the present application;

[0055] Figure 3This is a flowchart for implementing step S22 in the behavior prediction method based on AI digital human in one embodiment of the present application;

[0056] Figure 4 This is a flowchart for implementing step S30 in the behavior prediction method based on AI digital human in one embodiment of the present application;

[0057] Figure 5 This is a flowchart for implementing step S31 in the behavior prediction method based on AI digital human in one embodiment of the present application;

[0058] Figure 6 This is a flowchart for implementing step S40 in the behavior prediction method based on AI digital human in one embodiment of the present application;

[0059] Figure 7 This is a principle block diagram of a behavior prediction system based on AI digital human in one embodiment of the present application;

[0060] Figure 8 It is a schematic diagram of a device in one embodiment of the present application. DETAILED DESCRIPTION

[0061] The present application is further described in detail below with reference to the accompanying drawings.

[0062] In one embodiment, if Figure 1 As shown, this application discloses a behavior prediction method based on AI digital human, which specifically includes the following steps:

[0063] S10: Obtain raw visual data, voice data, and environmental sensor data, and perform preliminary preprocessing to obtain a processed data set.

[0064] Specifically, raw visual data, voice data and environmental sensor data are obtained, such as through cameras, microphones and various sensors. Visual data mainly includes user facial expressions, gestures and image data of the surrounding environment. Voice data includes the user's voice, voice content and tone changes. Environmental sensor data includes temperature, humidity, motion sensor and other information. After preliminary preprocessing, these data will be denoised, normalized and other processing, so that data from different sources can have a unified processing format, thus preparing for subsequent multimodal feature extraction and ensuring data consistency and accuracy.

[0065] S20: Extract multimodal features from the processed data set to obtain visual features, speech features, and environmental perception features, and perform weighted fusion of each modal feature based on the attention mechanism to generate a multimodal feature representation.

[0066] Specifically, visual features are used to extract the spatial features of images through convolutional neural networks, speech features are converted through spectrograms and then extracted through deep neural networks, and environmental perception features are used to extract key perception information through time series analysis of sensor data. When fusing each extracted modal feature, a weighted fusion method is adopted, focusing on the role and importance of each modal feature in a specific task. Based on the attention mechanism, the influence of different modalities in the behavior prediction process is adjusted to ensure that the effective information of each modality is fully utilized during the feature fusion process, and finally a comprehensive multimodal feature representation is generated.

[0067] S30: Based on multimodal feature representation, a deep learning model is used to optimize occlusion compensation, identify and repair occluded areas in the original visual data, and then generate an optimized feature set.

[0068] Specifically, a deep convolutional neural network is used to process the raw visual data. By analyzing the feature maps, possible occluded areas in the image are identified. The detected occluded areas are repaired by combining visual features and environmental data. An occlusion compensation algorithm is used to repair the occluded parts based on known information about the surrounding environment. For example, the occluded areas are optimized through a trained deep learning model, so that the occluded parts in the image are restored to the real scene as much as possible. Finally, an optimized feature set is generated for use in subsequent tasks.

[0069] S40: Use the time series analysis model to perform behavior prediction on the optimized feature set and generate behavior output related to the user's current interaction state.

[0070] Specifically, the optimized feature set serves as the input of the time series model. The time series analysis model identifies the temporal pattern of behavior by modeling time series data, captures the laws of user behavior changes over time, and predicts the behavioral status at future moments. The model not only considers the data at the current moment, but also combines historical data for analysis. Through time series feature extraction and modeling, it infers the user's possible future behavior and generates behavioral output related to the current interaction status based on the prediction results, accurately reflecting the user's real-time needs and behavioral characteristics.

[0071] In one embodiment, if Figure 2 As shown, in step S20, weighted fusion of each modality feature is performed based on the attention mechanism, specifically including:

[0072] S21: Design an independent weight matrix for each modal feature. The weight matrix is ​​designed based on the contribution of each modal feature in the behavior prediction task. The contribution is evaluated through the deep learning model during the training process, and an initial weight value is assigned to each modal feature based on the evaluation results.

[0073] Specifically, for each modal feature, such as visual features, speech features, and environmental perception features, an independent weight matrix is ​​designed. The design of the weight matrix is ​​based on the actual contribution of the modal feature in the behavior prediction task. The contribution evaluation is automatically completed during the training process through the deep learning model. The model will dynamically learn the influence of each modal feature on behavior prediction according to the requirements of the task. In this way, the model can assign an appropriate initial weight value to each modal feature to ensure that the role of each modality can be reasonably reflected in the subsequent feature fusion process.

[0074] S22: Based on the correlation and interactivity of each modal feature, a cross-modal attention mechanism is introduced to establish the interactive relationship between each modal feature, and the weight matrix of each modal feature is adjusted according to the interactive relationship.

[0075] Specifically, the cross-modal attention mechanism is used to capture the mutual influence and interaction between different modal features. By analyzing the correlation between visual, speech and environmental perception features, the cross-modal attention mechanism can determine which modal features have greater contributions in specific behavior prediction tasks, and adjust the weight matrix according to these interaction relationships, so that the influence of each modal feature in different situations can be optimized, thereby improving the overall prediction effect.

[0076] S23: Based on the adjusted weight matrix, the information within each modal feature is weightedly fused through the self-attention mechanism.

[0077] Specifically, after the cross-modal attention mechanism adjusts the weight matrix, the self-attention mechanism is used to perform weighted fusion on the information within each modal feature. By calculating the relationship between the elements within the modal features, it is determined which parts of the information are more critical to the final prediction results, so that the key features within each modal feature are strengthened, while redundant information is suppressed, thereby ensuring that the final multimodal feature representation is more refined and effective.

[0078] In one embodiment, if Figure 3 As shown, in step S22, based on the relevance and interactivity of each modal feature, a cross-modal attention mechanism is introduced to establish an interactive relationship between each modal feature, and the weight matrix of each modal feature is adjusted according to the interactive relationship, specifically including:

[0079] S221: Based on the correlation and interactivity of each modal feature, a cross-modal attention mechanism is constructed. By calculating the correlation between each modal feature, appropriate attention weights are assigned to each modal feature, thereby capturing the mutual influence relationship between different modalities.

[0080] Specifically, the correlation between the features of each modality is analyzed. By calculating the correlation between different modal features, it is possible to identify which modalities contribute more to the behavior prediction task. Based on this correlation information, a cross-modal attention mechanism is constructed and appropriate attention weights are assigned to each modality. The size of these weights depends on the importance of each modal feature in the task. By dynamically adjusting these weights, the contribution of each modality can be fully reflected, thereby effectively capturing the mutual influence relationship between different modal features and improving the model's processing capabilities and prediction accuracy for multimodal data.

[0081] S222: Based on the cross-modal attention mechanism, a multi-level attention module is used to process modal features layer by layer, capturing local and global modal interaction information at different levels.

[0082] Specifically, based on the cross-modal attention mechanism, a multi-level attention module is used to process the modal features layer by layer. The attention modules of each layer can not only capture local modal interaction information, but also capture the global interaction relationship through fusion of different levels, and understand and process the interaction between modalities from different scales. At each level, the attention mechanism will dynamically adjust the weights according to the output of the previous layer, so that the influence of each modal feature is further optimized, thereby ensuring that the processing of each level can effectively reflect the relationship between modalities.

[0083] S223: Based on modal interaction information, a cross-modal adaptive weighting mechanism is introduced to dynamically adjust the weight matrix according to the contribution of each modal feature.

[0084] Specifically, based on the interaction information between modal features, a cross-modal adaptive weighting mechanism is introduced. The weight matrix is ​​dynamically adjusted according to the actual contribution of each modal feature. Through in-depth analysis of the interaction relationship between modalities, the impact of each modality on the final prediction result is evaluated, and on this basis, the weight of each modal feature is dynamically adjusted. During the training process, the weight of the modal features is continuously optimized so that each modal feature plays the most appropriate role in the final decision-making process, thereby improving the overall prediction accuracy.

[0085] In one embodiment, if Figure 4 As shown, in step S30, based on the multimodal feature representation, a deep learning model is used to perform occlusion compensation optimization to identify and repair the occluded areas in the original visual data, specifically including:

[0086] S31: Based on the visual features in the multimodal feature representation, the occluded areas in the visual data are located through a deep convolutional neural network, and the contextual information and neighborhood features of the occluded areas are used to identify and calibrate the occluded areas.

[0087] Specifically, a deep convolutional neural network is used to accurately locate the occluded areas in the visual data, visual features are extracted through multi-layer convolution operations, and the occluded areas are identified in combination with the contextual information of the image. The calibration of the occluded areas not only depends on the feature information in the image, but also considers the neighborhood features around the occluded areas, so as to better understand the relationship between the occluded areas and other parts, and then accurately locate the position and range of the occluded areas, providing effective input data for subsequent repair steps.

[0088] S32: Combining speech features and environmental perception features, a cross-modal deep learning model is used for occlusion repair, and the compensation strategy of the occluded area is optimized through an adaptive learning mechanism.

[0089] Specifically, a cross-modal deep learning model is used to perform occlusion repair by combining the occluded areas and other modal features of the visual data, such as speech features and environmental perception features. The compensation strategy is dynamically adjusted according to the characteristics of the occluded area through an adaptive learning mechanism. Among them, the speech features and environmental perception features provide additional contextual information to help the model judge the background environment of the occluded area, thereby generating more reasonable repair content, and then accurately repairing the occlusion in the visual data, making the repaired content more natural and in line with the actual scene.

[0090] S33: Use recurrent neural networks to predict and optimize occlusion repair in time series. Based on the temporal characteristics of occlusion events, we can perform real-time repair of rapidly changing occluded areas in dynamic environments, and optimize the restoration effect of occluded areas based on the continuity of previous and next frames.

[0091] Specifically, a recurrent neural network is used to predict and optimize occlusion repair in time series data. By processing data in the time dimension, the temporal characteristics of occlusion events are captured, and the model is used to learn the changing patterns of occluded areas in dynamic environments, so as to perform real-time repair on rapidly changing occluded areas. In addition, by considering the continuity between previous and next frames, the temporal consistency of the repaired areas is ensured, thereby optimizing the restoration effect of the occluded areas and making the repair smoother and more natural.

[0092] S34: Refine the repaired occluded area based on the generative adversarial network and generate natural visual content through the generator.

[0093] Specifically, a generative adversarial network is used to further refine the repaired occluded area. The generator generates natural visual content consistent with the surrounding environment by learning the visual features of the repaired area, and generates realistic visual details based on the input image information, so that the repaired area is seamlessly integrated with the surrounding background, effectively improving the naturalness and realism of the occlusion repair, and avoiding the repaired area from appearing abrupt or inharmonious.

[0094] In one embodiment, if Figure 5 As shown, in step S31, the occluded area in the visual data is located by a deep convolutional neural network, and the context information and neighborhood features of the occluded area are used to identify and calibrate the occluded area, specifically including:

[0095] S311: Through multi-scale convolution operations, combined with the contextual information in the image, the surrounding environment features of the occluded area are captured.

[0096] Specifically, multi-scale convolution operations are used to process images, and convolution kernels of different scales are used to capture features at different levels in the image. The convolution operation at each scale can focus on contextual information in different ranges in the image, thereby extracting the surrounding environmental features of the occluded area at multiple scales. By combining the contextual information in the image, the relationship between the occluded area and its surrounding background can be better understood, thereby providing rich environmental features for subsequent occluded area positioning and repair, improving the accuracy of occluded area repair, and making the repaired image more natural and able to seamlessly blend with the surrounding unoccluded areas.

[0097] S312: Based on the neighborhood features of the occluded area, the spatial relationship between adjacent pixels is extracted, and then the boundary of the occluded area is calibrated to determine the position and range of the occluded area.

[0098] Specifically, by extracting the neighborhood features of the occluded area, further analyzing the spatial relationship between adjacent pixels, and analyzing the spatial relationship, the boundary of the occluded area is calibrated, and the position and range of the occluded area are determined. For example, through the local texture information in the image and the spatial layout of adjacent pixels, combined with the neighborhood features of the occluded area, the exact position of the occluded area is identified, and the range of its influence is determined, ensuring the overall consistency of the restoration effect with the image.

[0099] In one embodiment, if Figure 6 As shown, in step S40, the optimized feature set is predicted by the time series analysis model to generate a behavior output related to the user's current interaction state, specifically including:

[0100] S41: Based on the optimized feature set, by encoding the data in the time dimension, the pattern of interaction behavior evolving over time is captured to obtain time series features.

[0101] Specifically, based on the optimized feature set, by encoding the data in the time dimension, the evolution pattern of user interaction behavior over time is captured. By processing historical data, the patterns of behavioral changes are identified and converted into time series features to characterize the changes in user behavior at different time points, providing basic information for subsequent behavior predictions.

[0102] S42: Use a time series analysis model to model time series features, capture the impact of historical behavior on future behavior, and predict user behavior at the current moment and / or future moments.

[0103] Specifically, a time series analysis model is used to model the extracted time series features, capture the impact of historical behavior on future behavior, predict future behavior based on past behavior patterns, and predict the user's possible behavior at the current moment or in the future. By modeling the relationship between historical behavior and future behavior, more accurate behavior predictions are provided, thereby effectively supporting real-time decision-making.

[0104] S43: Generate behavior output related to the user's current interaction state based on the prediction results of the time series analysis model.

[0105] Specifically, based on the prediction results of the timing analysis model, behavioral outputs related to the user's current interaction status are generated to help understand the user's needs and intentions, thereby generating reasonable responses or actions, including information provided to the user, tasks performed, or system feedback, to ensure that the system can respond intelligently based on the user's current interaction status.

[0106] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0107] In one embodiment, a behavior prediction system based on AI digital human is provided, and the behavior prediction system based on AI digital human corresponds one-to-one with the behavior prediction method based on AI digital human in the above embodiment. Figure 7 As shown in Figure 1, the AI ​​digital human behavior prediction system includes a data acquisition and processing module, a feature extraction module, an occlusion compensation and optimization module, and a behavior prediction module. The detailed description of each functional module is as follows:

[0108] The data acquisition and processing module is used to obtain raw visual data, voice data, and environmental sensor data, and perform preliminary preprocessing to obtain a processed data set;

[0109] The feature extraction module is used to extract multimodal features from the processed data set to obtain visual features, speech features, and environmental perception features, and then perform weighted fusion of each modality feature based on the attention mechanism to generate a multimodal feature representation;

[0110] The occlusion compensation optimization module is used to optimize occlusion compensation based on multimodal feature representation and a deep learning model to identify and repair occluded areas in the original visual data, thereby generating an optimized feature set.

[0111] The behavior prediction module is used to perform behavior prediction on the optimized feature set through a time series analysis model and generate behavior output related to the user's current interaction state.

[0112] Optionally, the feature extraction module specifically includes:

[0113] The weight matrix design submodule is used to design an independent weight matrix for each modal feature. The weight matrix is ​​designed based on the contribution of each modal feature in the behavior prediction task. The contribution is evaluated by the deep learning model during the training process, and the initial weight value is assigned to each modal feature based on the evaluation results.

[0114] The cross-modal attention mechanism submodule is used to introduce a cross-modal attention mechanism based on the correlation and interactivity of each modal feature, establish the interactive relationship between each modal feature, and adjust the weight matrix of each modal feature according to the interactive relationship;

[0115] The self-attention weighted submodule is used to perform weighted fusion of the internal information of each modal feature through the self-attention mechanism based on the adjusted weight matrix.

[0116] Optionally, the weight matrix design submodule specifically includes:

[0117] The cross-modal attention calculation unit is used to build a cross-modal attention mechanism based on the correlation and interactivity of each modal feature. By calculating the correlation between the features of each modality, it assigns appropriate attention weights to each modal feature, thereby capturing the mutual influence relationship between different modalities.

[0118] Multi-level attention unit, based on the cross-modal attention mechanism, uses multi-level attention modules to process modal features layer by layer, capturing local and global modal interaction information at different levels;

[0119] The adaptive weighting unit is used to introduce a cross-modal adaptive weighting mechanism based on modal interaction information and dynamically adjust the weight matrix according to the contribution of each modal feature.

[0120] Optionally, the occlusion compensation optimization module specifically includes:

[0121] The occlusion region localization submodule is used to locate the occlusion region in the visual data through a deep convolutional neural network based on the visual features in the multimodal feature representation, and to identify and calibrate the occlusion region using the contextual information and neighborhood features of the occlusion region;

[0122] The cross-modal restoration submodule combines speech features and environmental perception features, uses a cross-modal deep learning model for occlusion restoration, and optimizes the compensation strategy for occluded areas through an adaptive learning mechanism.

[0123] The temporal repair prediction submodule uses a recurrent neural network to predict and optimize occlusion repair in time series. Based on the temporal characteristics of occlusion events, it performs real-time repair of rapidly changing occluded areas in dynamic environments and optimizes the restoration effect of occluded areas based on the continuity of previous and next frames.

[0124] The repair content generation submodule is used to refine the repaired occluded area based on the generative adversarial network and generate natural visual content through the generator.

[0125] Optionally, the occlusion area positioning submodule specifically includes:

[0126] Multi-scale convolution unit, used to capture the surrounding environment features of the occluded area by combining the context information in the image through multi-scale convolution operations;

[0127] The neighborhood relationship extraction unit is used to extract the spatial relationship between adjacent pixels based on the neighborhood features of the occlusion area, and then calibrate the boundary of the occlusion area and determine the position and range of the occlusion area.

[0128] Optionally, the behavior prediction module specifically includes:

[0129] The temporal feature encoding submodule is used to encode data in the time dimension based on the optimized feature set to capture the pattern of interaction behavior evolving over time and obtain temporal features;

[0130] The time series modeling submodule is used to model time series features using a time series analysis model, capture the impact of historical behavior on future behavior, and predict user behavior at the current moment and / or future moments;

[0131] The behavior output submodule is used to generate behavior output related to the user's current interaction status based on the prediction results of the time series analysis model.

[0132] For the specific limitations of the AI ​​digital human-based behavior prediction system, please refer to the limitations of the AI ​​digital human-based behavior prediction method above, and will not be repeated here. The various modules in the above-mentioned AI digital human-based behavior prediction system can be implemented in whole or in part through software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0133] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8As shown. The computer device includes a processor, memory, network interface and database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a behavior prediction method based on AI digital human is implemented.

[0134] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0135] Obtain raw visual data, voice data, and environmental sensor data, and perform preliminary preprocessing to obtain a processed data set;

[0136] Extract multimodal features from the processed dataset to obtain visual features, speech features, and environmental perception features, and perform weighted fusion of each modality feature based on the attention mechanism to generate a multimodal feature representation;

[0137] Based on multimodal feature representation, a deep learning model is used to optimize occlusion compensation, identify and repair occluded areas in the original visual data, and then generate an optimized feature set;

[0138] The optimized feature set is used to predict behavior through a time series analysis model to generate behavioral output related to the user's current interaction status.

[0139] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0140] Obtain raw visual data, voice data, and environmental sensor data, and perform preliminary preprocessing to obtain a processed data set;

[0141] Extract multimodal features from the processed dataset to obtain visual features, speech features, and environmental perception features, and perform weighted fusion of each modality feature based on the attention mechanism to generate a multimodal feature representation;

[0142] Based on multimodal feature representation, a deep learning model is used to optimize occlusion compensation, identify and repair occluded areas in the original visual data, and then generate an optimized feature set;

[0143] The optimized feature set is used to predict behavior through a time series analysis model to generate behavioral output related to the user's current interaction status.

[0144] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0145] Those skilled in the art will clearly understand that for the sake of convenience and brevity in description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

[0146] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A behavior prediction method based on AI digital human, characterized in that: The method comprises: Obtain raw visual data, voice data, and environmental sensor data, and perform preliminary preprocessing to obtain a processed data set; Extracting multimodal features from the processed data set to obtain visual features, speech features, and environmental perception features, and performing weighted fusion of the features of each modality based on an attention mechanism to generate a multimodal feature representation; Based on the multimodal feature representation, a deep learning model is used to perform occlusion compensation optimization, identify and repair occluded areas in the original visual data, and then generate an optimized feature set; Performing behavior prediction on the optimized feature set through a time series analysis model to generate behavior output related to the user's current interaction state; The weighted fusion of each modality feature based on the attention mechanism specifically includes: Designing an independent weight matrix for each modal feature, wherein the weight matrix is ​​designed based on the contribution of each modal feature to the behavior prediction task, the contribution is evaluated by a deep learning model during training, and initial weight values ​​are assigned to each modal feature based on the evaluation results; Based on the relevance and interactivity of the modal features, a cross-modal attention mechanism is introduced to establish an interactive relationship between the modal features, and the weight matrix of the modal features is adjusted according to the interactive relationship; The cross-modal attention mechanism is introduced based on the relevance and interactivity of the modal features, the interactive relationship between the modal features is established, and the weight matrix of the modal features is adjusted according to the interactive relationship, specifically including: Based on the relevance and interactivity of the modal features, a cross-modal attention mechanism is constructed. By calculating the correlation between the modal features, appropriate attention weights are assigned to the modal features, thereby capturing the mutual influence relationship between different modalities. Based on the cross-modal attention mechanism, a multi-level attention module is used to process modal features layer by layer, capturing local and global modal interaction information at different levels; Based on the modal interaction information, a cross-modal adaptive weighting mechanism is introduced to dynamically adjust the weight matrix according to the contribution of each modal feature; Based on the adjusted weight matrix, weighted fusion of the information within each modal feature is performed through a self-attention mechanism; The method of using a deep learning model to optimize occlusion compensation based on the multimodal feature representation to identify and repair occluded areas in the original visual data specifically includes: Based on the visual features in the multimodal feature representation, locating the occluded area in the visual data through a deep convolutional neural network, and using contextual information and neighborhood features of the occluded area to identify and demarcate the occluded area; Combining the speech features and the environmental perception features, a cross-modal deep learning model is used to perform occlusion repair, and a compensation strategy for the occluded area is optimized through an adaptive learning mechanism; Utilize recurrent neural networks to predict and optimize occlusion repair in time series. Based on the temporal characteristics of occlusion events, the system performs real-time repair of rapidly changing occluded areas in dynamic environments, and optimizes the restoration effect of occluded areas based on the continuity of previous and next frames. Refining the repaired occluded area based on a generative adversarial network, and generating natural visual content through a generator; The process of performing behavior prediction on the optimized feature set using a time series analysis model to generate behavior output related to the user's current interaction state specifically includes: Based on the optimized feature set, the data in the time dimension is encoded to capture the pattern of interaction behavior evolving over time, thereby obtaining time series features; A time series analysis model is used to model the time series features, capture the impact of historical behavior on future behavior, and predict user behavior at the current moment and / or future moments; Based on the prediction results of the time series analysis model, a behavior output related to the user's current interaction state is generated.

2. The behavior prediction method based on AI digital human according to claim 1, characterized in that: The method of locating the occluded area in the visual data by using a deep convolutional neural network and identifying and calibrating the occluded area by using contextual information and neighborhood features of the occluded area specifically includes: By combining the context information in the image with a multi-scale convolution operation, the surrounding environment features of the occluded area are captured; Based on the neighborhood features of the occluded area, the spatial relationship between adjacent pixels is extracted, and then the boundary of the occluded area is calibrated to determine the position and range of the occluded area.

3. A behavior prediction system based on AI digital human, characterized by: The AI ​​digital human-based behavior prediction system includes: The data acquisition and processing module is used to obtain raw visual data, voice data, and environmental sensor data, and perform preliminary preprocessing to obtain a processed data set; A feature extraction module is used to extract multimodal features from the processed data set to obtain visual features, speech features, and environmental perception features, and to perform weighted fusion of the features of each modality based on an attention mechanism to generate a multimodal feature representation; an occlusion compensation optimization module, configured to perform occlusion compensation optimization based on the multimodal feature representation using a deep learning model, identify and repair occluded areas in the original visual data, and generate an optimized feature set; A behavior prediction module, configured to perform behavior prediction on the optimized feature set using a time series analysis model, and generate behavior output related to the user's current interaction state; The feature extraction module specifically includes: A weight matrix design submodule is used to design an independent weight matrix for each modal feature. The weight matrix is ​​designed based on the contribution of each modal feature to the behavior prediction task. The contribution is evaluated by the deep learning model during training, and initial weight values ​​are assigned to each modal feature based on the evaluation results. A cross-modal attention mechanism submodule is used to introduce a cross-modal attention mechanism based on the relevance and interactivity of the modal features, establish an interactive relationship between the modal features, and adjust the weight matrix of the modal features according to the interactive relationship; A self-attention weighted submodule is used to perform weighted fusion of the information within each modal feature through a self-attention mechanism based on the adjusted weight matrix; The weight matrix design submodule specifically includes: The cross-modal attention calculation unit is used to construct a cross-modal attention model based on the relevance and interactivity of each modal feature. The attention mechanism calculates the correlation between the features of each modality and assigns appropriate attention weights to each modality feature, thereby Capture the mutual influence between different modalities; Multi-level attention unit, based on the cross-modal attention mechanism, uses multi-level attention modules layer by layer Process modal features to capture local and global modal interaction information at different levels; Adaptive weighting unit is used to introduce cross-modal adaptive weighting mechanism based on modal interaction information. The contribution of each modal feature dynamically adjusts the weight matrix; The occlusion compensation optimization module specifically includes: The occlusion region localization submodule is used to locate the occlusion region in the visual data through a deep convolutional neural network based on the visual features in the multimodal feature representation, and to identify and calibrate the occlusion region using the contextual information and neighborhood features of the occlusion region; The cross-modal restoration submodule combines speech features and environmental perception features, uses a cross-modal deep learning model for occlusion restoration, and optimizes the compensation strategy for occluded areas through an adaptive learning mechanism. The temporal repair prediction submodule uses a recurrent neural network to predict and optimize occlusion repair in time series. Based on the temporal characteristics of occlusion events, it performs real-time repair of rapidly changing occluded areas in dynamic environments and optimizes the restoration effect of occluded areas based on the continuity of previous and next frames. The inpainted content generation submodule is used to refine the inpainted occluded area based on the generative adversarial network and generate natural visual content through the generator; The behavior prediction module specifically includes: The temporal feature encoding submodule is used to encode data in the time dimension based on the optimized feature set to capture the pattern of interaction behavior evolving over time and obtain temporal features; The time series modeling submodule is used to model time series features using a time series analysis model, capture the impact of historical behavior on future behavior, and predict user behavior at the current moment and / or future moments; The behavior output submodule is used to generate behavior output related to the user's current interaction status based on the prediction results of the time series analysis model.

4. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the behavior prediction method based on AI digital human as described in any one of claims 1 to 2 are implemented.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the behavior prediction method based on AI digital human are implemented as described in any one of claims 1 to 2.

Images