AI digital human-based behavior prediction method, system and device, and storage medium

Abstract

The invention relates to a behavior prediction method, system and device based on an AI digital human, and a storage medium, and the method comprises the steps: obtaining original visual data, voice data and environment sensing data, and carrying out the preliminary preprocessing, and obtaining a processing data set; extracting multi-modal features from the processing data set to obtain visual features, voice features and environmental perception features, and performing weighted fusion on the modal features based on an attention mechanism to generate multi-modal feature representation; on the basis of multi-modal feature representation, a deep learning model is adopted for shielding compensation optimization, shielding areas in the original visual data are recognized and repaired, and then an optimized feature set is generated; and performing behavior prediction on the optimized feature set through a time sequence analysis model, and generating behavior output related to the current interaction state of the user. The method has the effect of improving the behavior prediction efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a behavior prediction method, system, device and storage medium based on AI digital human. Background Art

[0002] At present, with the rapid development of artificial intelligence technology, AI digital humans have been widely used in virtual reality, robot interaction, intelligent customer service and other fields. AI digital humans are virtual humans constructed through technologies such as computer vision, and have multiple interactive capabilities such as visual perception and speech recognition. In the task of behavior prediction, AI digital humans usually predict user behavior and reactions by processing and analyzing image data.

[0003] Existing behavior prediction methods mostly rely on single-modal data for modeling and processing, which can complete simple prediction tasks to a certain extent. However, due to the lack of full utilization of different modalities and the correlation and interactivity between modalities, the prediction accuracy is limited, especially in complex interaction scenarios, and cannot adapt to the dynamic changes of multimodal information.

[0004] The above-mentioned existing technical solutions have the following defects: the existing behavior prediction methods fail to fully consider the interactive relationship and dynamic changes between modes, resulting in low behavior prediction efficiency, so there is room for improvement. Summary of the invention

[0005] In order to improve the efficiency of behavior prediction, the present application provides a behavior prediction method, system, device and storage medium based on AI digital human.

[0006] The above-mentioned invention objective of the present application is achieved through the following technical solutions: A behavior prediction method based on AI digital human, the method comprising: Obtaining raw visual data, voice data, and environmental sensor data, and performing preliminary preprocessing to obtain a processed data set; Extracting multimodal features from the processed data set to obtain visual features, speech features, and environmental perception features, and weightedly fusing the features of each modality based on an attention mechanism to generate a multimodal feature representation; Based on the multimodal feature representation, a deep learning model is used to perform occlusion compensation optimization, identify and repair occluded areas in the original visual data, and then generate an optimized feature set; The optimized feature set is subjected to behavior prediction through a time series analysis model to generate behavior output related to the user's current interaction state.

[0007] By adopting the above technical solutions, by acquiring raw visual data, voice data and environmental sensor data and performing preliminary preprocessing, it is possible to integrate multiple information sources, improve the quality and integrity of the data, and thus provide richer and more accurate input data for subsequent feature extraction and analysis; by extracting multimodal features from the processed data set, visual features, voice features and environmental perception features are obtained, and weighted fusion of each modal feature is performed based on the attention mechanism, which can fully consider the importance of different modalities in behavior prediction and improve the accuracy and reliability of the prediction results; by using a deep learning model based on multimodal feature representation to optimize occlusion compensation, identify and repair occluded areas in the raw visual data, it is possible to overcome the impact of occlusion problems in the data and ensure that the model can operate normally in a complex environment, thereby improving the comprehensiveness and accuracy of the prediction; by using a timing analysis model to predict behavior on the optimized feature set, generating behavior output related to the user's current interaction state, it is possible to dynamically adjust behavior predictions, improve the real-time and personalization of system responses, and thus optimize the user interaction experience.

[0008] In one example, the present application may be further configured as follows: the weighted fusion of each modality feature based on the attention mechanism specifically includes: An independent weight matrix is ​​designed for each modal feature, wherein the weight matrix is ​​designed based on the contribution of each modal feature in the behavior prediction task, the contribution is evaluated by a deep learning model during training, and an initial weight value is assigned to each modal feature based on the evaluation result; According to the relevance and interactivity of the modal features, a cross-modal attention mechanism is introduced to establish an interactive relationship between the modal features, and the weight matrix of the modal features is adjusted according to the interactive relationship; Based on the adjusted weight matrix, the information within each modal feature is weightedly fused through a self-attention mechanism.

[0009] By adopting the above technical solution, by designing an independent weight matrix for each modal feature and designing it according to the contribution of the modal feature in the behavior prediction task, it is possible to accurately evaluate the impact of each modality on the final prediction result and avoid the interference of invalid features, thereby enhancing the accuracy and reliability of the prediction model; by introducing a cross-modal attention mechanism, it is possible to capture the interaction between different modalities and improve the depth and accuracy of feature fusion, thereby optimizing the comprehensiveness and accuracy of the prediction results.

[0010] In one example, the present application may be further configured as follows: based on the relevance and interactivity of the modal features, a cross-modal attention mechanism is introduced to establish an interactive relationship between the modal features, and the weight matrix of the modal features is adjusted according to the interactive relationship, specifically including: According to the relevance and interactivity of the features of each modality, a cross-modal attention mechanism is constructed, and by calculating the correlation between the features of each modality, appropriate attention weights are assigned to the features of each modality, thereby capturing the mutual influence relationship between different modalities; Based on the cross-modal attention mechanism, a multi-level attention module is used to process modal features layer by layer to capture local and global modal interaction information at different levels; Based on the modal interaction information, a cross-modal adaptive weighting mechanism is introduced to dynamically adjust the weight matrix according to the contribution of each modal feature.

[0011] By adopting the above technical solution and building a cross-modal attention mechanism based on the correlation and interactivity of each modal feature, it is possible to more finely explore the mutual influence relationship between different modalities, thereby improving the model's ability to fuse multimodal data; by processing modal features layer by layer through multi-level attention modules, capturing local and global modal interaction information at different levels, it is possible to deeply understand the complex relationship between modalities, thereby improving the model's ability to predict user behavior; by introducing an adaptive weighting mechanism, the weight matrix is ​​dynamically adjusted according to the contribution of each modal feature, which can flexibly adapt to changes in the importance of features in different scenarios, thereby further improving the accuracy and flexibility of prediction.

[0012] In one example, the present application may be further configured as follows: based on the multimodal feature representation, using a deep learning model to perform occlusion compensation optimization, identifying and repairing occluded areas in the original visual data, specifically including: Based on the visual features in the multimodal feature representation, locating the occluded area in the visual data through a deep convolutional neural network, and using the context information and neighborhood features of the occluded area to identify and mark the occluded area; Combining the speech features and the environmental perception features, a cross-modal deep learning model is used to perform occlusion repair, and a compensation strategy for the occluded area is optimized through an adaptive learning mechanism; The occlusion repair in the time series is predicted and optimized by using a recurrent neural network. According to the timing characteristics of the occlusion event, the occlusion area that changes rapidly in the dynamic environment is repaired in real time, and the restoration effect of the occlusion area is optimized according to the continuity of the previous and next frames. The repaired occluded area is refined based on a generative adversarial network, and natural visual content is generated through a generator.

[0013] By adopting the above technical solution, by using a deep convolutional neural network to locate the occluded area in the visual data based on the visual features in the multimodal feature representation, the occluded area can be efficiently and accurately identified and calibrated to avoid the occlusion problem affecting subsequent analysis; by combining speech features and environmental perception features, a cross-modal deep learning model is used for occlusion repair, which can use multiple modal information to compensate for occlusion and improve the naturalness and accuracy of the repair effect; by using a recurrent neural network to predict and optimize occlusion repair in time series, the occluded area that changes in a dynamic environment can be repaired in real time, thereby improving the system's adaptability and real-time performance to rapidly changing scenes; by refining the repaired occluded area through a generative adversarial network, more natural content that conforms to visual features can be generated, thereby improving the realism and visual quality of the repaired area.

[0014] In one example, the present application may be further configured as follows: locating the occluded area in the visual data by using a deep convolutional neural network, and identifying and calibrating the occluded area by using contextual information and neighborhood features of the occluded area, specifically including: By combining the context information in the image through multi-scale convolution operations, the surrounding environment features of the occluded area are captured; Based on the neighborhood features of the occluded area, the spatial relationship between adjacent pixels is extracted, and then the boundary of the occluded area is calibrated to determine the position and range of the occluded area.

[0015] By adopting the above technical solution, through multi-scale convolution operations and combining with the contextual information in the image, the environmental features around the occluded area can be effectively captured, thereby providing more comprehensive occluded area information and helping to improve the restoration quality; by extracting the spatial relationship between adjacent pixels based on the neighborhood features of the occluded area, the boundary and range of the occluded area can be accurately calibrated, the erroneous restoration area can be reduced, and the restoration accuracy and reliability can be improved.

[0016] In one example, the present application may be further configured as follows: the behavior prediction of the optimized feature set by the time series analysis model to generate a behavior output related to the user's current interaction state specifically includes: Based on the optimized feature set, the data in the time dimension is encoded to capture the pattern of interaction behavior evolving over time, thereby obtaining a time series feature; A time series analysis model is used to model the time series features, capture the impact of historical behaviors on future behaviors, and predict user behaviors at the current moment and / or at future moments; Based on the prediction results of the time series analysis model, a behavior output related to the user's current interaction state is generated.

[0017] By adopting the above technical solution and encoding the data in the time dimension, it is possible to capture the pattern of the evolution of interactive behavior over time, so as to understand the temporal laws of user behavior; by adopting a time series analysis model to model the time series characteristics, it is possible to better capture the impact of historical behavior on future behavior, thereby improving the accuracy of behavior prediction; by generating behavioral outputs related to the user's current interaction status based on the prediction results of the time series analysis model, it is possible to adjust the system response in a timely manner, optimize the user experience, and enhance the intelligence and personalization of the interaction.

[0018] The second object of the invention is achieved by the following technical solutions: A behavior prediction system based on AI digital human, the behavior prediction system based on AI digital human comprising: The data acquisition and processing module is used to obtain raw visual data, voice data and environmental sensor data, and perform preliminary preprocessing to obtain a processed data set; A feature extraction module is used to extract multimodal features from the processed data set to obtain visual features, speech features and environmental perception features, and to perform weighted fusion of each modality feature based on an attention mechanism to generate a multimodal feature representation; An occlusion compensation optimization module, configured to perform occlusion compensation optimization using a deep learning model based on the multimodal feature representation, identify and repair occluded areas in the original visual data, and generate an optimized feature set; The behavior prediction module is used to perform behavior prediction on the optimized feature set through a time series analysis model to generate a behavior output related to the user's current interaction state.

[0019] By adopting the above technical solutions, by acquiring raw visual data, voice data and environmental sensor data and performing preliminary preprocessing, it is possible to integrate multiple information sources, improve the quality and integrity of the data, and thus provide richer and more accurate input data for subsequent feature extraction and analysis; by extracting multimodal features from the processed data set, visual features, voice features and environmental perception features are obtained, and weighted fusion of each modal feature is performed based on the attention mechanism, which can fully consider the importance of different modalities in behavior prediction and improve the accuracy and reliability of the prediction results; by using a deep learning model based on multimodal feature representation to optimize occlusion compensation, identify and repair occluded areas in the raw visual data, it is possible to overcome the impact of occlusion problems in the data and ensure that the model can operate normally in a complex environment, thereby improving the comprehensiveness and accuracy of the prediction; by using a timing analysis model to predict behavior on the optimized feature set, generating behavior output related to the user's current interaction state, it is possible to dynamically adjust behavior predictions, improve the real-time and personalization of system responses, and thus optimize the user interaction experience.

[0020] The third objective of the present application is achieved through the following technical solutions: A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned behavior prediction method based on AI digital human when executing the computer program.

[0021] The fourth objective of the present application is achieved through the following technical solutions: A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned behavior prediction method based on AI digital humans.

[0022] In summary, this application includes the following beneficial technical effects: 1. By acquiring raw visual data, voice data and environmental sensor data and performing preliminary preprocessing, it is possible to integrate multiple information sources, improve the quality and integrity of the data, and thus provide richer and more accurate input data for subsequent feature extraction and analysis; by extracting multimodal features from the processed data set, visual features, voice features and environmental perception features are obtained, and weighted fusion of each modality feature is performed based on the attention mechanism, which can fully consider the importance of different modalities in behavior prediction and improve the accuracy and reliability of the prediction results; by using a deep learning model based on multimodal feature representation to optimize occlusion compensation, identify and repair occluded areas in the raw visual data, it is possible to overcome the impact of occlusion problems in the data and ensure that the model can operate normally in complex environments, thereby improving the comprehensiveness and accuracy of the prediction; by using a time series analysis model to predict the behavior of the optimized feature set, and generate behavior output related to the user's current interaction state, it is possible to dynamically adjust the behavior prediction, improve the real-time and personalization of the system response, and thus optimize the user interaction experience; 2. By designing independent weight matrices for each modal feature and designing them according to the contribution of the modal features in the behavior prediction task, the impact of each modality on the final prediction result can be accurately evaluated, and the interference of invalid features can be avoided, thereby enhancing the accuracy and reliability of the prediction model; by introducing a cross-modal attention mechanism, the interaction between different modalities can be captured, the depth and accuracy of feature fusion can be improved, and the comprehensiveness and accuracy of the prediction results can be optimized; 3. By constructing a cross-modal attention mechanism based on the correlation and interactivity of each modal feature, the mutual influence relationship between different modalities can be more finely explored, thereby improving the model's ability to integrate multimodal data; by processing modal features layer by layer through multi-level attention modules, local and global modal interaction information can be captured at different levels, which can deeply understand the complex relationship between modalities, thereby improving the model's ability to predict user behavior; by introducing an adaptive weighting mechanism, the weight matrix is ​​dynamically adjusted according to the contribution of each modal feature, which can flexibly adapt to changes in the importance of features in different scenarios, thereby further improving the accuracy and flexibility of prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of a behavior prediction method based on AI digital human in one embodiment of the present application; Figure 2 It is a flowchart for implementing step S20 in the behavior prediction method based on AI digital human in one embodiment of the present application; Figure 3 It is a flowchart for implementing step S22 in the behavior prediction method based on AI digital human in one embodiment of the present application; Figure 4 It is a flowchart for implementing step S30 in the behavior prediction method based on AI digital human in one embodiment of the present application; Figure 5 It is a flowchart for implementing step S31 in the behavior prediction method based on AI digital human in one embodiment of the present application; Figure 6 It is a flowchart for implementing step S40 in the behavior prediction method based on AI digital human in one embodiment of the present application; Figure 7 This is a principle block diagram of a behavior prediction system based on AI digital human in one embodiment of the present application; Figure 8 It is a schematic diagram of a device in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The present application is further described in detail below in conjunction with the accompanying drawings.

[0025] In one embodiment, if Figure 1 As shown, the present application discloses a behavior prediction method based on AI digital human, which specifically includes the following steps: S10: Acquire raw visual data, voice data, and environmental sensor data, and perform preliminary preprocessing to obtain a processed data set.

[0026] Specifically, raw visual data, voice data and environmental sensor data are obtained, such as through cameras, microphones and various sensors. Visual data mainly includes user facial expressions, gestures and image data of the surrounding environment. Voice data includes the user's voice, voice content and tone changes. Environmental sensor data includes temperature, humidity, motion sensor and other information. After preliminary preprocessing, these data will be denoised, normalized and other processing will be performed so that data from different sources can have a unified processing format, thus preparing for subsequent multimodal feature extraction and ensuring data consistency and accuracy.

[0027] S20: Extract multimodal features from the processed data set to obtain visual features, speech features, and environmental perception features, and perform weighted fusion of each modal feature based on the attention mechanism to generate a multimodal feature representation.

[0028] Specifically, visual features are used to extract spatial features of images through convolutional neural networks, speech features are converted through spectrograms and then extracted through deep neural networks, and environmental perception features are used to extract key perception information through time series analysis of sensor data. When fusing each extracted modal feature, a weighted fusion method is adopted, focusing on the role and importance of each modal feature in a specific task, and adjusting the influence of different modalities in the behavior prediction process based on the attention mechanism to ensure that the effective information of each modality is fully utilized during the feature fusion process, and finally generate a comprehensive multimodal feature representation.

[0029] S30: Based on multimodal feature representation, a deep learning model is used to optimize occlusion compensation, identify and repair occluded areas in the original visual data, and then generate an optimized feature set.

[0030] Specifically, a deep convolutional neural network is used to process the raw visual data, and possible occluded areas in the image are identified through analysis of feature maps. The detected occluded areas are repaired by combining visual features and environmental data. An occlusion compensation algorithm is used to repair the occluded parts based on known information about the surrounding environment. For example, the occluded areas are optimized through a trained deep learning model, so that the occluded parts in the image are restored to the real scene as much as possible. Finally, an optimized feature set is generated for use in subsequent tasks.

[0031] S40: Perform behavior prediction on the optimized feature set through a time series analysis model to generate behavior output related to the user's current interaction state.

[0032] Specifically, the optimized feature set is used as the input of the time series model. The time series analysis model identifies the time pattern of behavior by modeling the time series data, captures the law of user behavior changing over time, and predicts the behavior state at future moments. The model not only considers the data at the current moment, but also analyzes historical data. Through time series feature extraction and modeling, it infers the user's possible future behavior, and generates behavioral output related to the current interaction state based on the prediction results, accurately reflecting the user's real-time needs and behavioral characteristics.

[0033] In one embodiment, if Figure 2 As shown, in step S20, weighted fusion of each modality feature is performed based on the attention mechanism, specifically including: S21: An independent weight matrix is ​​designed for each modal feature. The weight matrix is ​​designed based on the contribution of each modal feature in the behavior prediction task. The contribution is evaluated through a deep learning model during the training process, and an initial weight value is assigned to each modal feature based on the evaluation results.

[0034] Specifically, for each modal feature, such as visual features, speech features, and environmental perception features, an independent weight matrix is ​​designed. The design of the weight matrix is ​​based on the actual contribution of the modal feature in the behavior prediction task. The contribution evaluation is automatically completed during the training process through the deep learning model. The model will dynamically learn the influence of each modal feature on behavior prediction according to the requirements of the task. In this way, the model can assign appropriate initial weight values ​​to each modal feature to ensure that the role of each modality can be reasonably reflected in the subsequent feature fusion process.

[0035] S22: Based on the correlation and interactivity of each modal feature, a cross-modal attention mechanism is introduced to establish the interactive relationship between each modal feature, and the weight matrix of each modal feature is adjusted according to the interactive relationship.

[0036] Specifically, the cross-modal attention mechanism is used to capture the mutual influence and interaction between different modal features. By analyzing the correlation between visual, speech and environmental perception features, the cross-modal attention mechanism can determine which modal features have a greater contribution in specific behavior prediction tasks, and adjust the weight matrix according to these interaction relationships, so that the influence of each modal feature in different situations can be optimized, thereby improving the overall prediction effect.

[0037] S23: Based on the adjusted weight matrix, the information within each modal feature is weightedly fused through the self-attention mechanism.

[0038] Specifically, after the cross-modal attention mechanism adjusts the weight matrix, the self-attention mechanism is used to perform weighted fusion on the information within each modal feature. By calculating the relationship between the elements within the modal features, it is determined which parts of the information are more critical to the final prediction results. The key features within each modal feature are strengthened, while redundant information is suppressed, thereby ensuring that the final multimodal feature representation is more refined and effective.

[0039] In one embodiment, if Figure 3 As shown, in step S22, based on the relevance and interactivity of each modal feature, a cross-modal attention mechanism is introduced to establish an interactive relationship between each modal feature, and the weight matrix of each modal feature is adjusted according to the interactive relationship, specifically including: S221: Based on the relevance and interactivity of each modal feature, a cross-modal attention mechanism is constructed. By calculating the correlation between each modal feature, appropriate attention weights are assigned to each modal feature, thereby capturing the mutual influence relationship between different modalities.

[0040] Specifically, the correlation between the features of each modality is analyzed. By calculating the correlation between the features of different modalities, it is possible to identify which modalities contribute more to the behavior prediction task. Based on this correlation information, a cross-modal attention mechanism is constructed and appropriate attention weights are assigned to each modality. The size of these weights depends on the importance of each modal feature in the task. By dynamically adjusting these weights, the contribution of each modality can be fully reflected, thereby effectively capturing the mutual influence relationship between different modal features and improving the model's processing capabilities and prediction accuracy for multimodal data.

[0041] S222: Based on the cross-modal attention mechanism, a multi-level attention module is used to process modal features layer by layer to capture local and global modal interaction information at different levels.

[0042] Specifically, based on the cross-modal attention mechanism, a multi-level attention module is used to process the modal features layer by layer. The attention modules of each layer can not only capture the local modal interaction information, but also capture the global interaction relationship through the fusion of different levels, and understand and process the interaction between modalities from different scales. At each level, the attention mechanism will dynamically adjust the weights according to the output of the previous layer, so that the influence of each modal feature is further optimized, thereby ensuring that the processing at each level can effectively reflect the relationship between modalities.

[0043] S223: Based on the modal interaction information, a cross-modal adaptive weighting mechanism is introduced to dynamically adjust the weight matrix according to the contribution of each modal feature.

[0044] Specifically, based on the interaction information between modal features, an adaptive cross-modal weighting mechanism is introduced. The weight matrix is ​​dynamically adjusted according to the actual contribution of each modal feature. Through in-depth analysis of the interaction relationship between modalities, the impact of each modality on the final prediction result is evaluated, and on this basis, the weight of each modal feature is dynamically adjusted. During the training process, the weight of the modal features is continuously optimized, so that each modal feature plays the most appropriate role in the final decision-making process, thereby improving the overall prediction accuracy.

[0045] In one embodiment, if Figure 4 As shown, in step S30, based on the multimodal feature representation, a deep learning model is used to perform occlusion compensation optimization, identify and repair the occluded area in the original visual data, specifically including: S31: Based on the visual features in the multimodal feature representation, the occluded areas in the visual data are located through a deep convolutional neural network, and the contextual information and neighborhood features of the occluded areas are used to identify and calibrate the occluded areas.

[0046] Specifically, the occluded areas in the visual data are accurately located through a deep convolutional neural network, visual features are extracted through multi-layer convolution operations, and the occluded areas are identified in combination with the contextual information of the image. The calibration of the occluded areas not only depends on the feature information in the image, but also considers the neighborhood features around the occluded areas, so as to better understand the relationship between the occluded areas and other parts, and then accurately locate the position and range of the occluded areas, providing effective input data for subsequent repair steps.

[0047] S32: Combining speech features and environmental perception features, a cross-modal deep learning model is used for occlusion repair, and the compensation strategy of the occluded area is optimized through an adaptive learning mechanism.

[0048] Specifically, a cross-modal deep learning model is used to perform occlusion repair by combining the occluded area and other modal features of the visual data, such as speech features and environmental perception features. The compensation strategy is dynamically adjusted according to the characteristics of the occluded area through an adaptive learning mechanism. Among them, speech features and environmental perception features provide additional contextual information to help the model judge the background environment of the occluded area, thereby generating more reasonable repair content, and then accurately repairing the occlusion in the visual data, making the repaired content more natural and in line with the actual scene.

[0049] S33: Use recurrent neural networks to predict and optimize occlusion repair in time series. According to the timing characteristics of occlusion events, fast-changing occlusion areas in dynamic environments are repaired in real time, and the restoration effect of occlusion areas is optimized according to the continuity of previous and next frames.

[0050] Specifically, a recurrent neural network is used to predict and optimize occlusion repair in time series data. By processing data in the time dimension, the timing characteristics of occlusion events are captured, and the changing pattern of the occluded area in a dynamic environment is learned through the model to perform real-time repair on the rapidly changing occluded area. In addition, by considering the continuity between the previous and next frames, the timing consistency of the repaired area is ensured, thereby optimizing the restoration effect of the occluded area and making the repair smoother and more natural.

[0051] S34: Refine the repaired occluded area based on the generative adversarial network and generate natural visual content through the generator.

[0052] Specifically, a generative adversarial network is used to further refine the repaired occluded area. The generator generates natural visual content consistent with the surrounding environment by learning the visual features of the repaired area, and generates realistic visual details based on the input image information, so that the repaired area is seamlessly integrated with the surrounding background, effectively improving the naturalness and realism of the occluded repair, and avoiding the repaired area from appearing abrupt or inharmonious.

[0053] In one embodiment, if Figure 5 As shown, in step S31, the occluded area in the visual data is located by a deep convolutional neural network, and the context information and neighborhood features of the occluded area are used to identify and calibrate the occluded area, which specifically includes: S311: Through multi-scale convolution operations, combined with contextual information in the image, the surrounding environment features of the occluded area are captured.

[0054] Specifically, a multi-scale convolution operation is used to process the image, and convolution kernels of different scales are used to capture features at different levels in the image. The convolution operation of each scale can focus on contextual information in different ranges in the image, thereby extracting the surrounding environmental features of the occluded area at multiple scales. By combining the contextual information in the image, the relationship between the occluded area and its surrounding background can be better understood, thereby providing rich environmental features for subsequent occluded area positioning and repair, improving the accuracy of occluded area repair, and making the repaired image more natural and able to seamlessly blend with the surrounding unoccluded areas.

[0055] S312: Based on the neighborhood features of the occluded area, the spatial relationship between adjacent pixels is extracted, and then the boundary of the occluded area is calibrated to determine the position and range of the occluded area.

[0056] Specifically, by extracting the neighborhood features of the occluded area, further analyzing the spatial relationship between adjacent pixels, and analyzing the spatial relationship, the boundary of the occluded area is calibrated, and the position and range of the occluded area are determined. For example, through the local texture information in the image and the spatial layout of adjacent pixels, combined with the neighborhood features of the occluded area, the exact position of the occluded area is identified, and the range of its influence is determined, ensuring the overall consistency of the restoration effect with the image.

[0057] In one embodiment, if Figure 6 As shown, in step S40, the behavior prediction is performed on the optimized feature set through the time series analysis model to generate the behavior output related to the user's current interaction state, which specifically includes: S41: Based on the optimized feature set, the data in the time dimension is encoded to capture the pattern of interaction behavior evolving over time and obtain the time series features.

[0058] Specifically, based on the optimized feature set, by encoding the data in the time dimension, the evolution pattern of user interaction behavior over time is captured. By processing historical data, the patterns of behavior changes are identified and converted into time series features to characterize the behavior changes of users at different time points, providing basic information for subsequent behavior predictions.

[0059] S42: Use a time series analysis model to model time series features, capture the impact of historical behaviors on future behaviors, and predict user behaviors at the current moment and / or future moments.

[0060] Specifically, a time series analysis model is used to model the extracted time series features, capture the impact of historical behavior on future behavior, predict future behavior based on past behavior patterns, predict the user's possible behavior at the current moment or in the future, and provide more accurate behavior predictions by modeling the relationship between historical behavior and future behavior, thereby effectively supporting real-time decision-making.

[0061] S43: Generate a behavior output related to the user's current interaction state based on the prediction result of the time series analysis model.

[0062] Specifically, based on the prediction results of the timing analysis model, behavioral outputs related to the user's current interaction status are generated to help understand the user's needs and intentions, thereby generating reasonable responses or actions, including information provided to the user, tasks performed, or system feedback, ensuring that the system can respond intelligently based on the user's current interaction status.

[0063] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0064] In one embodiment, a behavior prediction system based on AI digital human is provided, and the behavior prediction system based on AI digital human corresponds one-to-one with the behavior prediction method based on AI digital human in the above embodiment. Figure 7 As shown, the AI ​​digital human behavior prediction system includes a data acquisition and processing module, a feature extraction module, an occlusion compensation optimization module, and a behavior prediction module. The detailed description of each functional module is as follows: The data acquisition and processing module is used to obtain raw visual data, voice data and environmental sensor data, and perform preliminary preprocessing to obtain a processed data set; The feature extraction module is used to extract multimodal features from the processed data set to obtain visual features, speech features, and environmental perception features, and to perform weighted fusion of each modality feature based on the attention mechanism to generate a multimodal feature representation; The occlusion compensation optimization module is used to optimize occlusion compensation based on multimodal feature representation and deep learning models, identify and repair occluded areas in the original visual data, and then generate an optimized feature set; The behavior prediction module is used to perform behavior prediction on the optimized feature set through a time series analysis model and generate behavior output related to the user's current interaction state.

[0065] Optionally, the feature extraction module specifically includes: The weight matrix design submodule is used to design an independent weight matrix for each modal feature. The weight matrix is ​​designed based on the contribution of each modal feature in the behavior prediction task. The contribution is evaluated through the deep learning model during the training process, and the initial weight value is assigned to each modal feature based on the evaluation results. The cross-modal attention mechanism submodule is used to introduce the cross-modal attention mechanism based on the correlation and interactivity of each modal feature, establish the interactive relationship between each modal feature, and adjust the weight matrix of each modal feature according to the interactive relationship; The self-attention weighted submodule is used to perform weighted fusion of the information within each modal feature through the self-attention mechanism based on the adjusted weight matrix.

[0066] Optionally, the weight matrix design submodule specifically includes: The cross-modal attention calculation unit is used to construct a cross-modal attention mechanism based on the relevance and interactivity of each modal feature. By calculating the correlation between each modal feature, an appropriate attention weight is assigned to each modal feature, thereby capturing the mutual influence relationship between different modalities. Multi-level attention unit, which is used to process modal features layer by layer based on the cross-modal attention mechanism, using multi-level attention modules to capture local and global modal interaction information at different levels; The adaptive weighting unit is used to introduce a cross-modal adaptive weighting mechanism based on modal interaction information and dynamically adjust the weight matrix according to the contribution of each modal feature.

[0067] Optionally, the occlusion compensation optimization module specifically includes: The occluded region localization submodule is used to locate the occluded region in the visual data through a deep convolutional neural network based on the visual features in the multimodal feature representation, and to identify and calibrate the occluded region using the contextual information and neighborhood features of the occluded region; The cross-modal restoration submodule is used to combine speech features and environmental perception features, use a cross-modal deep learning model for occlusion restoration, and optimize the compensation strategy of the occluded area through an adaptive learning mechanism; The time series repair prediction submodule is used to predict and optimize the occlusion repair in the time series using a recurrent neural network. According to the time series characteristics of the occlusion event, the fast-changing occlusion area in the dynamic environment is repaired in real time, and the restoration effect of the occlusion area is optimized according to the continuity of the previous and next frames. The inpainted content generation submodule is used to refine the inpainted occluded area based on the generative adversarial network and generate natural visual content through the generator.

[0068] Optionally, the occlusion area positioning submodule specifically includes: A multi-scale convolution unit is used to capture the surrounding environment characteristics of the occluded area by combining the context information in the image through multi-scale convolution operations; The neighborhood relationship extraction unit is used to extract the spatial relationship between adjacent pixels based on the neighborhood features of the occluded area, and then calibrate the boundary of the occluded area to determine the position and range of the occluded area.

[0069] Optionally, the behavior prediction module specifically includes: The temporal feature encoding submodule is used to capture the evolution of interactive behaviors over time and obtain temporal features by encoding the data in the time dimension based on the optimized feature set; The time series modeling submodule is used to model time series features using a time series analysis model, capture the impact of historical behavior on future behavior, and predict user behavior at the current moment and / or future moments; The behavior output submodule is used to generate behavior output related to the user's current interaction state based on the prediction results of the time series analysis model.

[0070] For the specific limitations of the behavior prediction system based on AI digital humans, please refer to the limitations of the behavior prediction method based on AI digital humans mentioned above, which will not be repeated here. Each module in the above-mentioned behavior prediction system based on AI digital humans can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0071] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a behavior prediction method based on AI digital people is implemented.

[0072] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Obtaining raw visual data, voice data, and environmental sensor data, and performing preliminary preprocessing to obtain a processed data set; Extract multimodal features from the processed data set to obtain visual features, speech features, and environmental perception features, and perform weighted fusion of each modality feature based on the attention mechanism to generate a multimodal feature representation; Based on multimodal feature representation, a deep learning model is used to optimize occlusion compensation, identify and repair occluded areas in the original visual data, and then generate an optimized feature set; The optimized feature set is used to predict behavior through the time series analysis model to generate behavioral output related to the user's current interaction status.

[0073] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Obtaining raw visual data, voice data, and environmental sensor data, and performing preliminary preprocessing to obtain a processed data set; Extract multimodal features from the processed data set to obtain visual features, speech features, and environmental perception features, and perform weighted fusion of each modality feature based on the attention mechanism to generate a multimodal feature representation; Based on multimodal feature representation, a deep learning model is used to optimize occlusion compensation, identify and repair occluded areas in the original visual data, and then generate an optimized feature set; The optimized feature set is used to predict behavior through the time series analysis model to generate behavioral output related to the user's current interaction status.

[0074] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0075] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

[0076] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A behavior prediction method based on AI digital human, characterized in that: The method comprises: Obtaining raw visual data, voice data, and environmental sensor data, and performing preliminary preprocessing to obtain a processed data set; Extracting multimodal features from the processed data set to obtain visual features, speech features, and environmental perception features, and weightedly fusing the features of each modality based on an attention mechanism to generate a multimodal feature representation; Based on the multimodal feature representation, a deep learning model is used to perform occlusion compensation optimization, identify and repair occluded areas in the original visual data, and then generate an optimized feature set; The optimized feature set is subjected to behavior prediction through a time series analysis model to generate behavior output related to the user's current interaction state.

2. The behavior prediction method based on AI digital human according to claim 1 is characterized in that: The weighted fusion of each modality feature based on the attention mechanism specifically includes: An independent weight matrix is ​​designed for each modal feature, wherein the weight matrix is ​​designed based on the contribution of each modal feature in the behavior prediction task, the contribution is evaluated by a deep learning model during training, and an initial weight value is assigned to each modal feature based on the evaluation result; According to the relevance and interactivity of the modal features, a cross-modal attention mechanism is introduced to establish an interactive relationship between the modal features, and the weight matrix of the modal features is adjusted according to the interactive relationship; Based on the adjusted weight matrix, the information within each modal feature is weightedly fused through a self-attention mechanism.

3. The behavior prediction method based on AI digital human according to claim 2 is characterized in that: According to the relevance and interactivity of the modal features, a cross-modal attention mechanism is introduced to establish an interactive relationship between the modal features, and the weight matrix of the modal features is adjusted according to the interactive relationship, specifically including: According to the relevance and interactivity of the features of each modality, a cross-modal attention mechanism is constructed, and by calculating the correlation between the features of each modality, appropriate attention weights are assigned to the features of each modality, thereby capturing the mutual influence relationship between different modalities; Based on the cross-modal attention mechanism, a multi-level attention module is used to process modal features layer by layer to capture local and global modal interaction information at different levels; Based on the modal interaction information, a cross-modal adaptive weighting mechanism is introduced to dynamically adjust the weight matrix according to the contribution of each modal feature.

4. The behavior prediction method based on AI digital human according to claim 1 is characterized in that: The method of optimizing occlusion compensation by using a deep learning model based on the multimodal feature representation to identify and repair occluded areas in the original visual data specifically includes: Based on the visual features in the multimodal feature representation, locating the occluded area in the visual data through a deep convolutional neural network, and using the context information and neighborhood features of the occluded area to identify and mark the occluded area; Combining the speech features and the environmental perception features, a cross-modal deep learning model is used to perform occlusion repair, and a compensation strategy for the occluded area is optimized through an adaptive learning mechanism; The occlusion repair in the time series is predicted and optimized by using a recurrent neural network. According to the timing characteristics of the occlusion event, the occlusion area that changes rapidly in the dynamic environment is repaired in real time, and the restoration effect of the occlusion area is optimized according to the continuity of the previous and next frames. The repaired occluded area is refined based on a generative adversarial network, and natural visual content is generated through a generator.

5. The behavior prediction method based on AI digital human according to claim 4 is characterized in that: The method of locating the occluded area in the visual data by using a deep convolutional neural network and identifying and calibrating the occluded area by using contextual information and neighborhood features of the occluded area specifically includes: By combining the context information in the image through multi-scale convolution operations, the surrounding environment features of the occluded area are captured; Based on the neighborhood features of the occluded area, the spatial relationship between adjacent pixels is extracted, and then the boundary of the occluded area is calibrated to determine the position and range of the occluded area.

6. The behavior prediction method based on AI digital human according to claim 1, characterized in that: The method of performing behavior prediction on the optimized feature set through a time series analysis model to generate a behavior output related to the user's current interaction state specifically includes: Based on the optimized feature set, the data in the time dimension is encoded to capture the pattern of interaction behavior evolving over time, thereby obtaining a time series feature; A time series analysis model is used to model the time series features, capture the impact of historical behaviors on future behaviors, and predict user behaviors at the current moment and / or at future moments; Based on the prediction results of the time series analysis model, a behavior output related to the user's current interaction state is generated.

7. A behavior prediction system based on AI digital human, characterized in that: The behavior prediction system based on AI digital human includes: The data acquisition and processing module is used to obtain raw visual data, voice data and environmental sensor data, and perform preliminary preprocessing to obtain a processed data set; A feature extraction module is used to extract multimodal features from the processed data set to obtain visual features, speech features and environmental perception features, and to perform weighted fusion of each modality feature based on an attention mechanism to generate a multimodal feature representation; An occlusion compensation optimization module, configured to perform occlusion compensation optimization using a deep learning model based on the multimodal feature representation, identify and repair occluded areas in the original visual data, and generate an optimized feature set; The behavior prediction module is used to perform behavior prediction on the optimized feature set through a time series analysis model to generate a behavior output related to the user's current interaction state.

8. The AI ​​digital human behavior prediction system according to claim 7, characterized in that: The feature extraction module specifically includes: A weight matrix design submodule, used to design an independent weight matrix for each modal feature, wherein the weight matrix is ​​designed based on the contribution of each modal feature in the behavior prediction task, the contribution is evaluated by a deep learning model during training, and an initial weight value is assigned to each modal feature based on the evaluation result; A cross-modal attention mechanism submodule, used to introduce a cross-modal attention mechanism based on the relevance and interactivity of the modal features, establish an interactive relationship between the modal features, and adjust the weight matrix of the modal features according to the interactive relationship; The self-attention weighted submodule is used to perform weighted fusion of the information inside each modal feature through the self-attention mechanism based on the adjusted weight matrix.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the behavior prediction method based on AI digital human as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the behavior prediction method based on AI digital human as claimed in any one of claims 1 to 6 are implemented.

Images