A display interaction method and system

Abstract

This application discloses a display interaction method and system; relating to the field of image display technology, this application, after acquiring a video stream for an interactive object and identifying at least one video frame in the video stream to obtain the current emotion category of the interactive object, can determine the content to be displayed corresponding to the target display area, and perform content recognition on the content to be displayed to obtain the display attributes corresponding to the content to be displayed. Then, based on the current emotion category of the interactive object and the display attributes of the content to be displayed, the target attribute information and target posture information corresponding to the virtual object to be interacted with are determined. Subsequently, based on the target attribute information and target posture information, the virtual object to be interacted with is generated in the target display area, and the content to be displayed is broadcast and displayed on the target display area according to the virtual object to be interacted with. Therefore, the accuracy of information transmission can be improved, enabling users to intuitively understand the information represented by the data, thereby ensuring user experience.

Description

Technical Field

[0001] This invention relates to the field of image display technology, and more specifically to a display interaction method and system. Background Technology

[0002] In recent years, with the rapid development of internet technology, various smart terminal devices have been widely integrated into people's daily lives and work. Based on this, the information broadcasting and reminder functions of smart terminal devices have become increasingly rich, and users' demands for convenient and intuitive information access, as well as emotional and personalized interactive experiences, are growing. Especially in scenarios requiring centralized display and interaction of information, in order to provide users with more user-friendly and efficient interaction methods, the use of virtual character animation as a medium for display and interaction has gradually begun to be explored.

[0003] In the process of researching and practicing current technologies, the inventors of this application have found that related technologies typically respond to relevant data display on smart terminal devices only through simple text prompts or icon color changes. The interactive feedback is monotonous, and even if virtual images exist, they are only used as decorative elements. As a result, the accuracy of information transmission is insufficient, the information represented by the data cannot be intuitively understood, and the user experience is poor. Summary of the Invention

[0004] This application provides a display interaction method and related devices. The related devices may include a display interaction system, electronic devices, computer program products, and computer-readable storage media, which can improve the accuracy of information transmission, enable users to intuitively understand the information represented by the data, and thus ensure user experience.

[0005] A display interaction method, applied to a target display area of ​​a display terminal, includes: Acquire a video stream for the interactive object, and identify at least one video frame in the video stream to obtain the current emotion category of the interactive object; Determine the content to be displayed corresponding to the target display area, and perform content recognition on the content to be displayed to obtain the display attributes corresponding to the content to be displayed; Based on the current emotion category of the interactive object and the display attributes of the content to be displayed, determine the target attribute information and target posture information corresponding to the virtual object to be interacted with; Based on the target attribute information and the target posture information, the virtual object to be interacted with is generated in the target display area, and the content to be displayed is broadcast and displayed on the target display area according to the virtual object to be interacted with.

[0006] Accordingly, embodiments of this application provide a display interaction system, including: An information acquisition unit is used to acquire a video stream for an interactive object and to identify at least one video frame in the video stream to obtain the current emotion category of the interactive object. A content recognition unit is used to determine the content to be displayed corresponding to the target display area, and to perform content recognition on the content to be displayed in order to obtain the display attributes corresponding to the content to be displayed; The information determination unit is used to determine the target attribute information and target posture information corresponding to the virtual object to be interacted with, based on the current emotion category of the interactive object and the display attributes of the content to be displayed. The display interaction unit is used to generate the virtual object to be interacted with in the target display area based on the target attribute information and the target posture information, and to broadcast and display the content to be displayed in the target display area according to the virtual object to be interacted with.

[0007] Furthermore, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to execute the display interaction method provided in embodiments of this application.

[0008] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the display interaction methods provided in embodiments of this application.

[0009] Furthermore, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the display interaction method provided in embodiments of this application.

[0010] This embodiment of the application, after acquiring a video stream for an interactive object and identifying at least one video frame in the video stream to obtain the current emotion category of the interactive object, can determine the content to be displayed corresponding to the target display area, and perform content recognition on the content to be displayed to obtain the display attributes corresponding to the content to be displayed. Then, based on the current emotion category of the interactive object and the display attributes of the content to be displayed, the target attribute information and target posture information corresponding to the virtual object to be interacted with are determined. Subsequently, based on the target attribute information and target posture information, the virtual object to be interacted with is generated in the target display area, and the content to be displayed is broadcast and displayed on the target display area according to the virtual object to be interacted with. Because this scheme can dynamically determine the target attribute information and target posture information of the virtual object according to the current emotion category of the interactive object and the display attributes of the content to be displayed, the performance of the virtual object is matched with both the user's emotional state and the urgency of the broadcast content, thereby improving the accuracy of information transmission, enabling users to intuitively understand the information represented by the data, and thus ensuring user experience. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of a scenario for the display interaction method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the display interaction method provided in an embodiment of this application; Figure 3 This is a schematic diagram of a display area provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of the display interaction system provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] This application provides a display interaction method and related equipment, which may include a display interaction system, an electronic device, a computer program product, and a computer-readable storage medium. The display interaction system may be integrated into an electronic device, which may be a server or a terminal, etc.

[0015] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0016] For example, see Figure 1 Taking the integration of a display interaction system into an electronic device as an example, the electronic device can acquire a video stream targeting the interactive object and identify at least one video frame in the video stream to obtain the current emotion category of the interactive object. Then, it can determine the content to be displayed corresponding to the target display area and perform content recognition on the content to be displayed to obtain the corresponding display attributes. Next, based on the current emotion category of the interactive object and the display attributes of the content to be displayed, it can determine the target attribute information and target posture information corresponding to the virtual object to be interacted with. Finally, based on the target attribute information and target posture information, it can generate the virtual object to be interacted with in the target display area and broadcast and display the content to be displayed on the target display area according to the virtual object. Therefore, it can improve the accuracy of information transmission, enabling users to intuitively understand the information represented by the data, thereby ensuring a good user experience.

[0017] It is understood that, in the specific embodiments of this application, data such as video streams of interactive objects and content to be displayed are involved. When the following embodiments of this application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0018] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0019] This embodiment will be described from the perspective of a display interaction system, which can be integrated into an electronic device, such as a server or a terminal. The terminal can include tablet computers, laptops, personal computers (PCs), wearable devices, virtual reality devices, or other smart devices capable of display interaction.

[0020] like Figure 2 As shown, Figure 2 This is a schematic flowchart of an optional display interaction method provided in an embodiment of the present disclosure. The display interaction method can be applied to a target display area of ​​a display terminal, and the specific flow of the display interaction method can be as follows: 101. Obtain the video stream for the interactive object, and identify at least one video frame in the video stream to obtain the current emotion category of the interactive object.

[0021] In this context, the interactive object can be understood as the user who views the target display area in front of the display terminal and interacts with the virtual object. The video stream can be understood as a sequence of image frames continuously captured by a camera unit on the display terminal at a preset sampling frequency. A video stream may include at least one video frame. The preset sampling frequency can be set according to the hardware capabilities of the actual terminal device and the application scenario; for example, it can be set between 5Hz and 30Hz.

[0022] The current emotion category can be understood as the dominant emotion type exhibited by the interactive object within the current time window. This could include one of the basic emotion categories such as neutral, happy, sad, angry, fearful, surprised, or disgusted. The current emotion category reflects the real-time emotional state of the interactive object before or at the time of watching the virtual object's broadcast content.

[0023] There are several ways to acquire the video stream of the interactive object, including: for example, capturing the video stream in real-time using the built-in camera module of the display terminal; acquiring the video stream through an external camera device connected to the display terminal; reading the video stream from a pre-recorded video file; or acquiring the video stream from a remote camera device via network transmission, and so on. In practical applications, the specific method for acquiring the video stream can be flexibly selected based on the hardware configuration of the terminal device and the application scenario.

[0024] Optionally, in some implementations, after acquiring the video stream, at least one video frame in the video stream can be identified to obtain the current emotion category of the interactive object. The identification method may specifically include steps S11 to S14: S11. Extract the multimodal data of the interactive object from at least one video frame of the video stream.

[0025] Multimodal data can be understood as various types of data extracted from video streams that reflect the emotional state of interacting objects. Multimodal data can include at least one modality, and modal data can be understood as a single-source or single-form data type. For example, facial expression data extracted from video frames, body movement data extracted from video frames, audio data corresponding to video frames captured by a microphone, and text data corresponding to video frames obtained from a text input interface can all be considered as one type of multimodal data.

[0026] There are several ways to extract multimodal data of the interactive object from at least one video frame of the video stream, such as: for example, facial region localization can be performed on the video frame, and then the facial region can be cropped and normalized to obtain facial expression image data; or, the coordinate sequence of human body key points can be extracted from the video frame to obtain limb movement data; or, the microphone of the display terminal can be called simultaneously to collect the voice signal of the interactive object as voice modal data; or, the text input of the interactive object can be obtained from the text input interface of the display terminal as text modal data, and so on.

[0027] It should be noted that in practical applications, not all modal data must be used for every emotion recognition. When terminal device hardware is limited or the interacting object does not generate data for a certain modality, only a portion of the available modal data can be used for subsequent emotion recognition. However, to ensure the robustness and accuracy of emotion recognition, this application's embodiments preferentially employ at least two modal data for fusion recognition.

[0028] S12. Extract features from each modal data in the multimodal data to obtain the modal features of each modal data.

[0029] Modal features can be understood as high-dimensional vector representations of emotion-related information in modal data, obtained after feature extraction. Different modal data correspond to different feature extraction methods, and the extracted modal features reside in their respective semantic feature spaces.

[0030] Specifically, the method of extracting features from each modal data in the multimodal data to obtain the modal features of each modal data can include: (1) When the modal data is facial expression image data, features can be extracted from it based on the facial expression feature extraction network in the feature extraction model to obtain the modal features corresponding to the facial expression image data, that is, the facial expression category probability distribution vector: the network uses a lightweight convolutional neural network (such as MobileNetV3) as the backbone network, followed by a fully connected layer and a softmax classification layer. Inputting the normalized facial expression image into the network can obtain the facial expression category probability distribution vector. For the t-th video frame, its facial expression category probability distribution vector can be expressed by the following formula (1): in, It can represent the probability distribution vector of facial expression categories in frame t, with dimension K; K is the total number of predefined emotion categories, K≥3, and in this embodiment, K=7, corresponding to the seven basic emotions of "neutral, happy, sad, angry, fear, surprise, and disgust"; This can represent the probability value that the facial expression of the interactive object in frame t belongs to the k-th emotion category, satisfying... ∈[0,1] and =1, which is a dimensionless probability value; It can represent the normalized facial region image of frame t; It can be represented as a facial expression feature extraction network; [*]T can represent the vector transpose sign.

[0031] (2) When the modal data is limb movement data, features can be extracted from it based on the limb movement feature extraction network in the feature extraction model to obtain the modal features corresponding to the limb movement data, i.e., limb movement modal features: This network is based on a lightweight pose estimation algorithm to extract the human body key point coordinate sequence from video frames. Let the number of human body key points be J (e.g., in the embodiment of this application, J=17, covering the head, shoulder, elbow, wrist, hip, knee, ankle, etc.), and the two-dimensional coordinates of each key point are ( , ),in, and Let x and y be the x and y coordinates of the j-th keypoint in frame t, respectively, in a normalized coordinate system (dimensionless, ranging from [0,1]). Then, based on the keypoint coordinate sequence, the displacement of the same keypoint between adjacent frames can be calculated to obtain the limb movement amplitude, and thus the limb motion modal features can be obtained.

[0032] (3) When the modal data is speech data, the speech data can be analyzed for emotion by the speech emotion recognition network in the feature extraction model to obtain the speech emotion score as the speech modal feature.

[0033] (4) When the modal data is text data, the text content in the text data can be analyzed for emotion through the text emotion recognition network in the feature extraction model, and the text emotion score can be used as the text modal feature.

[0034] It should be noted that the emotion analysis in (3) and (4) can be understood as the process of identifying the emotional tendency contained in speech data or text content and quantifying this emotional tendency into a continuous emotion score. Specifically, for speech emotion analysis, the speech emotion recognition network can extract acoustic features such as prosodic features, pitch features, energy features, and speech rate features from the speech data and map these acoustic features into an emotion score; for text emotion analysis, the text emotion recognition network can perform semantic understanding of the text content, extract keywords and semantic structures that express emotions, and map this semantic information into an emotion score. The range of the emotion score is usually [0,1], with a larger value indicating a more positive and stronger emotion.

[0035] Furthermore, in this embodiment, facial expression modality features and body movement modality features are directly extracted from video frames, belonging to the visual channel; while speech modality features and text modality features are derived from the audio channel and text channel, respectively. Modality features from different channels are uniformly fused through an attention mechanism in subsequent steps to fully utilize the complementarity of multi-channel information and improve the accuracy and robustness of emotion recognition.

[0036] S13. Use a feature extraction model to determine the modal features and attention weights corresponding to each modal data to obtain the weighted modal features.

[0037] The feature extraction model can be understood as a neural network model used to extract and fuse emotion-related feature information from multimodal data. In this embodiment, the feature extraction model can be a multimodal emotion recognition model, which may include multiple sub-networks (such as a facial expression feature extraction network, a body movement feature extraction network, a speech emotion recognition network, and a text emotion recognition network), each used to process input data from different modalities, and can achieve weighted fusion of multimodal features through an attention mechanism. The attention weight can be understood as a coefficient used to characterize the degree of contribution of different modal features to the final emotion recognition result; the larger the attention weight, the higher the importance of the corresponding modal feature in the current recognition task.

[0038] Specifically, the feature extraction model uses modal features and corresponding attention weights for each modal data to obtain weighted modal features, which may include the following steps: (1) Attention fusion of facial expression modal features over time is performed to obtain the fused emotion category distribution vector: Since facial expression modal features (i.e., facial expression category probability distribution vectors) correspond to each video frame, multiple facial expression category probability distribution vectors can exist within a time window. To eliminate noise and jitter in single-frame emotion recognition, it is necessary to fuse the facial expression modal features in the time dimension. Specifically, first, obtain the facial expression category probability distribution vectors of all frames within the time window, and calculate the mean vector of these vectors. The calculation method of the mean vector can be expressed by the following formula (2): in, It can be a dimensionless mean vector representing the probability distribution vectors of facial expressions across all frames within a time window. It can represent the total number of frames within the current time window; This can represent the probability distribution vector of facial expression categories in frame t (S12 output).

[0039] After obtaining the mean vector, the temporal attention weight of each frame within the time window can be calculated based on the mean vector. The calculation method of the temporal attention weight can be expressed by the following formula (3): in, It can represent the temporal attention weight of the t-th frame (dimensionless); It can represent the traversal index (dimensionless integer) in the summation operation, and is used to traverse and sum all frames in the time window from frame 1 to frame T one by one; It can represent the probability distribution vector of facial expression categories in the i-th frame during the traversal process; It can represent the inner product operation of two vectors, and the result of the inner product is a scalar (dimensionless). It can characterize the temperature hyperparameter (dimensionless) and be used to control the sharpness of the attention distribution. In this embodiment, the empirical value τ = 0.1 is used. (*) can characterize an exponential function with the natural constant e as its base.

[0040] After calculating the temporal attention weight for each frame, the facial expression category probability distribution vectors of all frames within the time window can be weighted and summed based on the attention weights to obtain the fused emotion category distribution vector. The fusion calculation method can be expressed by the following formula (4): in, It can represent the distribution vector of the fused emotion categories, with a dimension of K.

[0041] After the above processing, the facial expression modality features are fused from the probability distribution vectors of multiple frames into a unified emotion category distribution vector, which will be used as the facial expression modality in subsequent intermodal attention weighting.

[0042] (2) After completing the temporal fusion of facial expression modal features, it is necessary to map the features of each modality to a unified feature dimension in order to perform intermodal attention weighting. Specifically: For facial expression modal features, the user emotion category distribution vector obtained after time-dimensional fusion has a dimension of K (total number of emotion categories). This vector can be directly used as the unified dimensional feature vector for facial expression modal features. For body movement modal features, the amplitude of body movement is a scalar, which can be mapped to dimension K through a fully connected layer to obtain the unified dimensional feature vector for body movement modal features. For speech modal features, the speech emotion score can be a scalar, which can also be mapped to dimension K through a fully connected layer to obtain the unified dimensional feature vector for speech modal features. For text modal features, the text emotion score can be a scalar, which can also be mapped to dimension K through a fully connected layer to obtain the unified dimensional feature vector for text modal features.

[0043] (3) Calculate the intermodal attention weights based on the unified dimension feature vectors of each modality to obtain weighted modal features. Specifically: Input the unified dimension feature vectors of each modality into the attention network, calculate the attention weights corresponding to each modal feature through the attention network, and then multiply the attention weights corresponding to each modal feature element by element with the unified dimension feature vector of that modality to obtain the weighted modal features of each modal data.

[0044] It's important to note that the attention network can be a lightweight, fully connected network. Its input is a feature vector of a uniform dimension for each modality, and its output is the attention weight value corresponding to each modality. After softmax normalization, the attention weight values ​​are such that the sum of all weights is 1. In multimodal emotion recognition, different modal features may contribute differently to emotion judgment. For example, when the facial expression of an interactive object is relatively blurry but the emotional features in the speech signal are obvious, the attention weight of the speech modality should be increased accordingly, while the attention weight of the facial expression modality should be decreased accordingly. By introducing an attention mechanism, the contribution ratio of each modality feature can be adaptively adjusted, improving the accuracy of emotion recognition.

[0045] S14. The weighted modal features are fused to obtain the emotion distribution data of the interactive object, and the current emotion category of the interactive object is determined based on the emotion distribution data.

[0046] Among them, emotion distribution data can be understood as the probability distribution vector of the interactive object on each predefined emotion category, which reflects the probability of the interactive object being in each type of emotional state.

[0047] The method of fusing weighted modal features to obtain the emotion distribution data of the interactive object, and determining the current emotion category of the interactive object based on the emotion distribution data, may specifically include the following steps: (1) The weighted modal features of each modality are concatenated or summed to obtain a fusion feature vector; the fusion feature vector is input into the classification layer (which can be a fully connected layer with a softmax activation function) to obtain the emotion distribution data of the interactive object.

[0048] When only the facial expression modality exists, the fusion feature vector is the weighted modal feature of the facial expression modality itself, and the emotion distribution data is the emotion category distribution vector obtained after the time dimension fusion in S13. When multiple modalities exist, the fusion feature vector is the concatenation result of the weighted modal features of each modality, and the classification layer outputs the final emotion distribution data based on the concatenated fusion feature vector.

[0049] (2) After obtaining the emotion distribution data, the current emotion category of the interactive object can be determined based on the emotion distribution data. Specifically, each component in the emotion distribution data can be compared, and the component with the largest value can be found. The emotion category corresponding to this component can be determined as the current emotion category of the interactive object. The determination of the current emotion category can be expressed by the following formula (5): in, It can represent the current emotion category of the interacting object; It can represent the k-th component in the emotion distribution data, representing the probability value that the current emotion of the interacting object belongs to the k-th emotion category. k can represent the index of the emotion category, with a value range of [1,K]; K can represent the total number of defined emotion categories; It can be characterized by taking such that The operation to obtain the maximum value of k involves iterating through all possible values ​​of k and finding the value that maximizes the maximum value. The largest one, k.

[0050] (3) After determining the current emotion category, the confidence level of the current emotion category can also be obtained. Specifically, the probability value corresponding to the current emotion category is extracted from the emotion distribution data, and this probability value is used as the confidence level of the current emotion category. The confidence level can be calculated using the following formula (6): in, It can represent the confidence level of the current emotion category, with a value range of [0,1]. The higher the confidence level, the more reliable the recognition result of the current emotion category. It can represent taking all The operation to find the maximum value in the middle is to iterate through all possible values ​​of k and find the value that makes k the maximum value in the middle. The largest one, k.

[0051] (4) After obtaining the current emotion category and confidence level, the emotion intensity of the interactive object can also be calculated (a continuous scalar value used to participate in the calculation of dynamic weights and the determination of multi-partition refresh priority in subsequent steps). The calculation process is as follows: First, calculate the visual emotion score based on the confidence level and the current emotion category. That is, first find the corresponding number in the predefined emotion valence sequence table according to the current emotion category. This number reflects the high and low position of the current emotion category in the emotion valence dimension; then, divide the number by the maximum value of the emotion valence number to obtain the normalized emotion value; finally, multiply the normalized emotion value by the confidence level to obtain the visual emotion score. The calculation method of the visual emotion score can be expressed by the following formula (7): in, It can represent the original visual emotion score, with a value range of [0,1]. It can represent the index of the current emotion category in the predefined emotion valence sequence list, with a value range of [0, K-1]. The sequence is arranged from low to high emotion valence, and the larger the index, the higher the emotion valence. K can represent the total number of emotion categories; K-1 can represent the maximum value of the emotion valence index, which is used to normalize the emotion valence index to [0,1]. It can characterize the normalized emotional efficacy value.

[0052] (5) The speech emotion score output by the speech emotion recognition network in step S12 and the text emotion score output by the text emotion recognition network in step S12 can be obtained. The text emotion score, speech emotion score and the above-mentioned visual emotion score are respectively subjected to Z-score standardization to eliminate the differences in the dimensions and distributions of the emotion scores of each modality. The calculation method of the standardization process can be expressed by the following formula (8). in, It can represent the standardized sentiment score of the j-th modality; An index that can represent the modal type Where t represents the text modality, a represents the speech modality, and v represents the visual modality; It can represent the raw sentiment score of the j-th modality. When j=t, it represents the text sentiment score; when j=a, it represents the speech sentiment score; and when j=v, it represents the visual sentiment score. It can represent the mean of the emotion score of the j-th modality over historical data; It can represent the standard deviation of the emotion score of the j-th modality over historical data.

[0053] (6) The standardized emotion scores of the text, speech, and visual modalities are linearly weighted according to the preset modal weight coefficients to obtain the emotion intensity of the interactive object after multimodal fusion. Specifically, the standardized emotion score of the text modal is multiplied by the weight coefficient of the text modal, the standardized emotion score of the speech modal is multiplied by the weight coefficient of the speech modal, and the standardized emotion score of the visual modal is multiplied by the weight coefficient of the visual modal. The three products are then added together to obtain the emotion intensity of the interactive object. The calculation method of the emotion intensity of the interactive object can be expressed by the following formula (9): in, It can characterize the emotional intensity of the interacting object; the larger the value, the more positive and intense the emotion. Standardized sentiment scores can characterize text modalities; Standardized emotion scores that can characterize speech modalities; Standardized emotion scores that can characterize visual modalities; Weighting coefficients that can characterize text modalities; Weighting coefficients that can characterize speech modalities; Weighting coefficients can characterize visual modalities.

[0054] It should be noted that, in this embodiment, the data output in step S14 may include: the current emotion category, confidence level, user emotion category distribution vector (i.e., emotion distribution data), and the emotion intensity of the interactive object. The current emotion category and user emotion category distribution vector are used to map into emotion semantic vectors in subsequent steps to participate in the construction of fusion features and the calculation of emotion matching degree in posture selection; the emotion intensity of the interactive object is used to determine the multi-partition refresh priority in subsequent steps; and the confidence level is used to characterize the reliability of the emotion recognition result. The above output data characterizes the emotional state of the interactive object from different dimensions, providing complete emotional dimension information for the generation and adjustment of virtual objects, and also providing a user-side input basis for determining the display attributes corresponding to the content to be displayed in subsequent step 102.

[0055] 102. Determine the content to be displayed corresponding to the target display area, and perform content recognition on the content to be displayed to obtain the display attributes corresponding to the content to be displayed.

[0056] The target display area can be understood as a specific functional area on the display screen of the display terminal used to display virtual objects and broadcast content to be displayed. In this embodiment, the display area of ​​the display terminal can be divided into multiple functional areas according to a preset layout, such as a virtual avatar display area, a content display area, an information overview area, and a to-do list area. The virtual avatar display area is the target display area in this step. This area is used to display the animated screen of the virtual object to be interacted with, and is the core display area for the virtual object to broadcast information and interact with the user.

[0057] The content to be displayed can be understood as the information that needs to be broadcast and displayed to the interactive object through the virtual object to be interacted with in the target display area. The content to be displayed can include various text information that needs to be conveyed to the user through the virtual object, such as weather warnings, environmental monitoring information, schedule reminders, news information, government announcements, and financial push notifications. Display attributes can be understood as attribute parameters used to characterize the urgency or priority of the content to be displayed when it is shown. Display attributes reflect the level of urgency at which the content to be displayed needs to be conveyed to the user. For example, taking weather warnings as an example, when the content to be displayed is "Red Rainstorm Warning," its corresponding display attribute has a high level of urgency; when the content to be displayed is "Tomorrow's Weather is Sunny," its corresponding display attribute has a low level of urgency.

[0058] Specifically, determining the content to be displayed corresponding to the target display area and performing content recognition on the content to be displayed to obtain the display attributes corresponding to the content to be displayed can be achieved through the following steps S21 to S24: S21: Obtain the content to be displayed corresponding to the target display area from the display terminal, extract the semantics of the content to be displayed to obtain the content semantic features, and classify the content semantic features corresponding to the content to be displayed according to their urgency to obtain the content urgency corresponding to the content semantic features.

[0059] Content semantic features can be understood as vector representations obtained by semantically encoding the textual information of the content to be displayed. These vector representations characterize the semantic meaning of the content to be displayed and serve as an important basis for subsequent content urgency classification and feature fusion construction. Content urgency can be understood as an urgency value determined based on the textual semantic information of the content to be displayed. This urgency value reflects the degree of urgency or warning implied in the textual expression of the content to be displayed.

[0060] There are several ways to obtain the content to be displayed corresponding to the target display area from the display terminal, including: for example, obtaining the content to be displayed from an application running on the display terminal, such as a weather application, calendar reminder application, news application, or government information application; or obtaining the content to be displayed from a network server connected to the display terminal, such as a weather warning platform server, environmental monitoring platform server, or information push server; or reading the content to be displayed from the pre-configured broadcast plan from the local storage space of the display terminal; or obtaining the content to be displayed pushed by an external system from the real-time communication interface of the display terminal, and so on.

[0061] After obtaining the content to be displayed, semantic extraction can be performed to obtain semantic features. The semantic extraction method can be as follows: First, the text information of the content to be displayed can be segmented into word sequences. Then, the word sequences are input into a pre-trained text semantic encoder (such as a lightweight variant of BERT or RoBERTa). This text semantic encoder can semantically encode the text information of the content to be displayed, outputting a fixed-dimensional original semantic vector. After obtaining the original semantic vector, L2 normalization can be performed to normalize the magnitude of the original semantic vector to 1, thus obtaining the content semantic features. The purpose of L2 normalization is to eliminate the influence of different text lengths on the magnitude of the semantic vector, so that the similarity result is not affected by the vector magnitude when calculating semantic similarity.

[0062] Specifically, the text information of the content to be displayed can be represented as X = [ , ,..., ],in, H can represent the i-th word in the text sequence, where H is the total number of words. After inputting the text information of the content to be displayed into the text semantic encoder, the original semantic vector output by the text semantic encoder can be represented by the following formula (10): in, It can represent the original semantic vector output by the text semantic encoder, with a dimension of D (D is the semantic vector dimension, and in this embodiment, D=256), which is a real number vector; It can characterize a text semantic encoder; It can represent the word sequence corresponding to the text information of the content to be displayed.

[0063] After obtaining the original semantic vector, L2 normalization can be performed on the original semantic vector to obtain the content semantic features. The calculation method of L2 normalization can be expressed by the following formula (11): in, It can represent the semantic features of the content after L2 normalization. It is a dimensionless real number vector with a magnitude of 1. The L2 norm operation can characterize a vector, which is the square root of the sum of the squares of all elements in the vector. The result of the L2 norm operation is a non-negative scalar, and it is consistent with... Having the same dimensions, therefore molecules With denominator The result after division It is a dimensionless vector with consistent dimensions.

[0064] After obtaining the semantic features of the content, the urgency of the content to be displayed can be classified according to these semantic features to obtain the content urgency level. The classification method can be as follows: an urgency classifier can be used to classify the urgency level. This urgency classifier can be a binary classification network with a fully connected layer and a softmax activation function. Its input is the semantic features of the content, and its output is the probability value of the content belonging to the "urgent" category. After inputting the semantic features of the content into the urgency classifier, the urgency classifier outputs the probability value of the content belonging to the "urgent" category. This probability value is used as the content urgency level. express.

[0065] It should be noted that the content urgency reflects the degree of urgency inherent in the textual expression of the content to be displayed. For example, the text content "earthquake warning" usually has a high content urgency after being classified by the urgency classifier; while the text content "today's weather is sunny" usually has a low content urgency after being classified by the urgency classifier.

[0066] S22: Acquire multi-source environmental sensor data, perform urgency level normalization mapping on each environmental sensor data, and obtain the environmental sensor urgency level.

[0067] Multi-source environmental sensing data can be understood as data collected from multiple environmental sensors from different sources, used to reflect the current environmental state. Multi-source environmental sensing data can include at least one environmental sensor data point, which can include various environmental monitoring data such as PM2.5 concentration data, PM10 concentration data, seismic intensity data, temperature anomaly data, humidity anomaly data, and wind speed anomaly data. For example, taking PM2.5 concentration data as an example, when the measured PM2.5 concentration is significantly higher than the safety threshold, it indicates that the current air quality is poor and the environmental sensing urgency is high. Environmental sensing urgency can be understood as a value obtained from a comprehensive assessment of environmental sensing data, used to reflect the degree of urgency of the current environmental state.

[0068] The method for acquiring multi-source environmental sensor data and normalizing the urgency level of each environmental sensor data point to obtain the environmental sensor urgency level can be as follows: First, acquire multi-source environmental sensor data through environmental sensors set on the display terminal or an environmental monitoring platform connected to the display terminal; then, for each type of environmental sensor data, obtain its corresponding lower limit of safety threshold and upper limit of emergency threshold; next, normalize the current measured value of each type of environmental sensor data, linearly mapping the current measured value to the [0,1] interval to obtain the normalized urgency level value corresponding to that type of environmental sensor data; finally, take the maximum value among the normalized urgency levels corresponding to all types of environmental sensor data as the environmental sensor urgency level.

[0069] Specifically, the normalization mapping method for each type of environmental sensing data can be expressed by the following formula (12): in, It can characterize the urgency of environmental sensing, with a value range of [0,1]. The larger the value, the higher the degree of environmental urgency. It can represent the type sequence number of environmental sensing data; It can represent the total number of types of environmental sensing data; It can characterize the current measured value of the q-th type of environmental sensor data and has physical dimensions (e.g., the unit of PM2.5 concentration is μg / m³). 3 (Earthquake intensity is measured in degrees, and temperature is measured in degrees Celsius). This can characterize the lower limit of the safety threshold for the q-th type of environmental sensing data, and... Having the same physical dimensions, when ≤ At the time, it was believed that the indicator posed no urgent risk; This can characterize the upper limit of the emergency threshold for the q-th type of environmental sensing data, and... Having the same physical dimensions, when ≥ At that time, the urgency level of this indicator was considered to be the highest. (*) can represent a cutoff function, used to restrict the input value to the interval [0,1]; {*} can represent the operation of maximizing the normalized urgency value corresponding to Q-type environmental sensing data; in this formula, the numerator... With denominator All have the same For physical quantities with the same dimensions, dividing the two yields a dimensionless ratio, indicating that the dimensions are consistent.

[0070] It should be noted that by normalizing and mapping each type of environmental sensor data separately and taking the maximum value, it can be ensured that when any type of environmental sensor data is in an emergency, the urgency of the environmental sensing can be reflected in a timely manner, avoiding the masking of the emergency status of individual indicators due to the averaging of multiple types of environmental data.

[0071] S23: Obtain external early warning data corresponding to the preset platform and map the external early warning data to the urgency of external information.

[0072] The "pre-set platform" can be understood as a pre-connected third-party information service platform used to provide external early warning information. Pre-set platforms can include various authoritative information release platforms such as meteorological early warning platforms, earthquake early warning platforms, geological disaster early warning platforms, and public safety early warning platforms. External early warning data can be understood as early warning information data released by authoritative institutions and obtained from the pre-set platform. External early warning data can include meteorological early warning level data, earthquake early warning level data, geological disaster early warning level data, and public safety incident early warning level data. For example, taking meteorological early warning level data as an example, a "red alert for heavy rain" issued by a meteorological early warning platform corresponds to a high warning level, and the resulting external information urgency is also high. External information urgency can be understood as a quantitative value extracted from external early warning data, reflecting the assessment of the urgency of the current event by an external authoritative institution.

[0073] The method of obtaining external early warning data corresponding to a preset platform and mapping the external early warning data to an external information urgency level can be as follows: First, obtain external early warning data from the preset platform through a network interface connected to the display terminal; then, according to the preset early warning level mapping rules, map the early warning level information carried in the external early warning data to an urgency value in the range [0,1] to obtain the external information urgency level, which can then be used... express.

[0074] It should be noted that the warning levels in external early warning data are usually represented in the form of discrete levels (such as blue warning, yellow warning, orange warning, red warning, or level 1 warning, level 2 warning, level 3 warning, etc.). Through the preset warning level mapping rules, warning information of different levels can be uniformly mapped to continuous external information urgency values, which facilitates subsequent weighted fusion with content urgency and environmental sensing urgency.

[0075] S24: The urgency of the content, the urgency of the environmental sensors, and the urgency of the external information are weighted and integrated according to preset weights to obtain the display attributes corresponding to the content to be displayed.

[0076] The preset weights can be understood as pre-defined weight coefficients used to characterize the contribution ratio of content urgency, environmental sensing urgency, and external information urgency to the overall display attributes. The preset weights satisfy the condition that the sum of all weight coefficients is 1 and that all weight coefficients are non-negative.

[0077] The method of weighting and fusing content urgency, environmental sensing urgency, and external information urgency according to preset weights to obtain the display attribute corresponding to the content to be displayed can be as follows: First, obtain the content urgency output in step S21, the environmental sensing urgency output in step S22, and the external information urgency output in step S23; then, multiply the content urgency by its corresponding weight coefficient, the environmental sensing urgency by its corresponding weight coefficient, and the external information urgency by its corresponding weight coefficient; finally, add the three products together, truncate the result, and use the truncated value as the display attribute corresponding to the content to be displayed.

[0078] Specifically, the calculation method for the display attribute can be expressed by the following formula (13): in, It can characterize the display attributes corresponding to the content to be displayed. It is a dimensionless scalar with a value range of [0,1]. The larger the value, the higher the urgency of the content to be displayed. It can represent the urgency of the content, is a dimensionless scalar, and takes values ​​in the range of [0,1]. It is output by step S21. It can characterize the urgency of environmental sensing, is a dimensionless scalar, and takes values ​​in the range of [0,1]. It is output by step S22. It can characterize the urgency of external information, is a dimensionless scalar, and takes values ​​in the range of [0,1]. It is output by step S23. This can represent the preset weight coefficient corresponding to the urgency of the content; It can represent the preset weighting coefficients corresponding to the urgency of environmental sensing; It can represent the preset weight coefficients corresponding to the urgency of external information.

[0079] It should be noted that after obtaining the display attributes, the display attributes can be further discretized to obtain discrete display level levels. Specifically, the display level can be calculated based on the display attributes and the total number of preset level divisions. The calculation method of the display level can be expressed by the following formula (14): in, It can represent the display level, with a value range of [1, L]; It can represent the total number of preset level divisions; in this embodiment of the application, L=5. It can represent the rounding up operation.

[0080] It should also be noted that the data output in step 102 may include: the display attributes, display level, and content semantic features corresponding to the content to be displayed. The display attributes and display level can be used in the subsequent step 103 for dynamic weight calculation and urgency compatibility judgment in posture selection; the content semantic features can be used in subsequent step 103 for weighted fusion with emotion semantic features to construct fused features. The above output data provides a complete description of the urgency level for the generation and adjustment of virtual objects from the content side, and together with the user emotion data output in step 101, constitutes the dual-input basis for the subsequent generation of virtual objects.

[0081] 103. Based on the current emotion category of the interactive object and the display attributes of the content to be displayed, determine the target attribute information and target posture information of the virtual object to be interacted with.

[0082] The target attribute information can be understood as information describing the appearance and voice performance of the virtual object to be interacted with when broadcasting content. Target attribute information can include facial parameters, voice parameters, and configuration parameters. Facial parameters can be understood as parameters controlling the facial expressions of the virtual object to be interacted with, including parameters such as mouth opening degree, mouth corner angle, eyebrow height, and eye opening degree. Voice parameters can be understood as prosody parameters controlling the speech of the virtual object to be interacted with, including parameters such as speech rate, pitch frequency, and volume. Configuration parameters can be understood as parameters controlling the display colors of the virtual object to be interacted with and the broadcast interface, including RGBA values ​​of parameters such as virtual character clothing color, background color, text color, and highlight color.

[0083] Target pose information can be understood as information describing the body movements performed by the virtual object to be interacted with when displaying content. Target pose information can include pose nodes and pose parameters. A pose node can be understood as a controlled joint node identifier in the skeletal model of the virtual object to be interacted with, and pose parameters can be understood as parameter values ​​such as rotation angle and movement speed corresponding to each pose node. For example, taking the pose node as the "right elbow joint node," its corresponding pose parameters can include the target rotation angle and target movement speed of the joint, used to control the range and speed of the virtual object's right arm movement.

[0084] It should be noted that in step 103, the generation of target attribute information and the filtering of target pose information are two parallel processing chains. Both chains rely on the fusion features output in step S32 as their core basis: the target attribute information is directly regressed and output based on the fusion features through the parameterized generation network in the preset generation model; the target pose information is obtained by similarity filtering and parameter modulation based on the fusion features through the preset pose library corresponding to the preset generation model. The outputs of the two chains together constitute the final output of step 103.

[0085] Specifically, the method for determining the target attribute information and target posture information of the virtual object to be interacted with, based on the current emotion category of the interactive object and the display attributes of the content to be displayed, can be as follows: steps S31~S34: S31: Extract semantic features from the current emotion category and display attribute to obtain the emotion features corresponding to the current emotion category and the display attribute features corresponding to the display attribute.

[0086] The emotion feature corresponding to the current emotion category can be understood as a vector representation obtained by mapping the current emotion category to a continuous semantic space. This vector representation can characterize the semantic meaning of the current emotion category. The display attribute feature corresponding to the display attribute can be understood as a vector representation obtained by mapping the display attribute to a continuous semantic space. This vector representation can characterize the urgency of the content to be displayed in the semantic space.

[0087] Specifically, the method of extracting semantic features from the current emotion category and display attribute to obtain the emotion features corresponding to the current emotion category and the display attribute features corresponding to the display attribute can be as follows: For the current emotion category, obtain the user emotion category distribution vector output in step S13. The user's emotion category distribution vector is mapped to an emotion semantic vector to obtain the emotion features corresponding to the current emotion category. Specifically, each emotion category is predefined to correspond to a pre-trained emotion semantic embedding vector. The emotion semantic embedding vectors of all emotion categories are weighted and summed according to the probability values ​​in the emotion category distribution vector to obtain the emotion semantic vector. The calculation method of the emotion semantic vector can be expressed by the following formula (15): in, It can represent the emotional features (i.e., emotional semantic vector) corresponding to the current emotional category. It is a real vector of dimension D, where D is the semantic vector dimension. K can represent the index of the emotion category; K can represent the total number of predefined emotion categories. It can represent the k-th component in the emotion category distribution vector, that is, the probability value that the current emotion of the interactive object belongs to the k-th emotion category; The pre-trained semantic embedding vector that can represent the k-th type of emotion is a real vector of dimension D, which is a fixed parameter vector obtained by training a large-scale emotion corpus.

[0088] For display attributes, obtain the semantic features of the content output in step S21. The semantic features of the content are directly used as the display attribute features corresponding to the display attributes. The semantic features of the content have already been obtained in step 102 through text semantic encoder and L2 normalization, and will not be calculated again here.

[0089] It should be noted that emotional characteristics and display attributes Both are located in a continuous semantic space of dimension D and have undergone L2 normalization. The magnitude of both is 1, which ensures that the two feature vectors are consistent in numerical scale during the subsequent weighted fusion in step S32, and that the fusion result will not be biased towards one side due to the difference in magnitude.

[0090] S32: Weighted fusion of emotional features and display attribute features to obtain fused features.

[0091] The fusion feature can be understood as a comprehensive semantic feature vector obtained by weighting and fusing the emotional features corresponding to the current emotion category and the display attribute features corresponding to the display attribute according to dynamically determined weights. The fusion feature simultaneously encodes the emotional semantic information of the interactive object and the content semantic information of the content to be displayed.

[0092] The weighted fusion of emotional features and display attribute features to obtain the fused features can be achieved as follows: (1) Calculate the semantic weight of the content based on the display attributes: Obtain the display attributes output in step S24 of step 102. Based on the display attribute, the content semantic weight is calculated using a nonlinear mapping function. The content semantic weight represents the proportion of the display attribute feature in the fused features. The characteristics of the nonlinear mapping function are: when the display attribute approaches 0, the content semantic weight approaches the lower limit; when the display attribute approaches 1, the content semantic weight approaches the upper limit; in the intermediate region, the content semantic weight increases smoothly with the increase of the display attribute. The content semantic weight can be calculated using the following formula (16): in, It can represent the semantic weight of content; It can represent the lower limit of the semantic weight of content; It can represent the upper limit of the semantic weight of content; It can characterize the curvature parameter of weight adjustment, which is used to control how quickly the semantic weight of content increases with display attributes; It can represent the base of the natural logarithm; It can characterize the core terms of a nonlinear mapping function, when When =0, this term is 0; when When =1, this term is 1.

[0093] (2) Calculate the sentiment weight based on the content semantic weight: After calculating the content semantic weight, calculate the sentiment weight of the interactive object. The sentiment weight represents the proportion of the sentiment feature in the fused features. The sentiment weight can be calculated using the following formula (17): in, It can represent the weight of emotions.

[0094] (3) Adjust the weights based on the display level threshold constraint: Obtain the display level output in step S24 of step 102. Based on the display level, the semantic weight of the content and the emotional weight of the interactive object are adjusted by threshold constraints to obtain the adjusted content weight and emotional weight. The purpose of the threshold constraint adjustment is: when the display level reaches the high level threshold, the content weight is forced to be raised to no less than the preset high level content weight threshold to ensure that the semantic content dominates in emergency scenarios; when the display level is lower than or equal to the low level threshold, the content weight is forced to be reduced to no more than the preset low level content weight threshold to ensure that the user's emotion dominates in non-emergency scenarios. The calculation method of the adjusted content weight can be expressed by the following formula (18): in, It can represent the adjusted content weight; It can represent the adjusted sentiment weight; It can represent the semantic weight of content; It can represent the display level; This can represent a high-level threshold, and a value of 4 is acceptable. This can represent a low-level threshold, which can be 2; The minimum threshold that can characterize the content weight in high-level scenarios (scenarios where the urgency of the content to be displayed is high) can be 0.8; The highest threshold that can represent the content weight in low-level scenarios (ordinary scenarios where the urgency of the content to be displayed is low) can be 0.3.

[0095] (4) Weighted fusion of emotional features and display attribute features based on adjusted content weights and emotional weights: After obtaining the adjusted content weights and emotional weights, the emotional features and display attribute features output in step S31 are weighted and fused based on these two weights. Specifically, the emotional features are multiplied by the emotional weights, the display attribute features are multiplied by the content weights, and the two products are added together to obtain the fused semantic vector. The calculation method of the fused semantic vector can be expressed by the following formula (19): in, It can represent the fused semantic vector; It can represent the adjusted sentiment weight; It can represent the emotional characteristics corresponding to the current emotion category; It can represent the adjusted content weight; It can characterize the display attribute features corresponding to the display attributes.

[0096] (5) Perform L2 normalization on the fused semantic vector to obtain the fused features: After obtaining the fused semantic vector, perform L2 normalization on the fused semantic vector to obtain the normalized fused features. The purpose of L2 normalization is to make the magnitude of the fused features equal to 1, so that the similarity calculation is not affected by the magnitude of the vector. The calculation method of L2 normalization can be expressed by the following formula (20): in, It can characterize the fusion features after L2 normalization.

[0097] It should be noted that the fusion feature output in this step simultaneously encodes the emotional semantics of the interactive object and the content semantics of the content to be displayed, and the contribution ratio of the two is dynamically adjusted by the display attributes. When the urgency of the display attributes is low, the emotional weight dominates, and the fusion feature reflects more the emotional state of the interactive object; when the urgency of the display attributes is high, the content weight dominates, and the fusion feature reflects more the urgency semantics of the content to be displayed. Through this dynamic weighted fusion mechanism, a smooth transition can be achieved between user emotional alignment and urgency content delivery. The fusion feature can also serve as input for both the parameterized generation of target attribute information in step S33 and the filtering of target pose information in step S34.

[0098] S33: Using a preset generation model, parameterized generation is performed based on fusion features to obtain target attribute information.

[0099] The preset generation model can be understood as a neural network model used to generate target attribute information and target pose information of the virtual object to be interacted with based on the fused features. The preset generation model can include multiple sub-networks, such as a facial parameter generation network, a speech prosody parameter generation network, a configuration parameter generation network, and a preset pose library. For example, taking the expression parameter generation network as an example, this network can be a multilayer perceptron network, whose input is the fused features and the emotion category distribution vector, and whose output is the facial parameter vector.

[0100] Specifically, the method of obtaining target attribute information by using a preset generation model and parametric generation based on fused features can be as follows: (1) Parameterization of facial parameters: Obtain the fusion features output in step S32 And the emotion category distribution vector output in step S13 The two data points mentioned above are input into the facial parameter generation network in the preset generation model. The facial parameters are generated by regression through the facial parameter generation network, and the facial parameters are output. The facial parameter generation network can be a multilayer perceptron network, which contains multiple fully connected layers and nonlinear activation function layers. It can map the fused features and emotion category distribution vector into a continuous parameter space of facial parameters. The facial parameters can be generated by the following formula (21): in, It can characterize facial parameters, with dimensions of , This represents the total number of facial parameters. (*) can characterize the facial parameter generation network.

[0101] Afterwards, the facial parameter generation network can... and After concatenation, the data is input into a fully connected layer, mapped layer by layer to the facial parameter space, and finally outputs the facial parameters. Because... and All vectors are dimensionless, and there is no issue of inconsistent dimensions when they are concatenated and input into the network for layer-by-layer mapping.

[0102] (2) Parameterized generation of speech parameters: Obtain the semantic features of the content output in step S21. and the display attributes output in step S24 The two data points are input into the speech prosody parameter generation network in the preset generation model. The speech parameters are then regressed and generated through the speech prosody parameter generation network to output a speech parameter vector. The speech parameter vector may include speech rate parameters, pitch fundamental frequency parameters, and volume parameters. Among them, the speech rate parameter and the display attributes satisfy a specific numerical relationship. Specifically, a basic speech rate is first generated through the speech prosody parameter generation network, and then the basic speech rate is adjusted according to the display attributes to obtain the final speech rate. The final speech rate can be calculated using the following formula (22): in, It can characterize the speech rate of the final synthesized speech, in units of words per second; It can represent the basic speech rate, with the unit being words per second; It can represent the speech rate adjustment coefficient, indicating the percentage increase in speech rate relative to the base speech rate for every unit increase in the displayed attribute.

[0103] The generation method of speech parameters can be represented by the following formula (23): in, It can characterize the generated speech parameters, with dimensions of , This represents the total number of voice parameters. (*) can characterize the speech prosody parameter generation network.

[0104] (3) Parameterization of configuration parameters: Obtain the fusion features output in step S32 and the display attributes output in step S24 The above two data are input into the configuration parameter generation network in the preset generation model. The configuration parameters are generated by regression through the configuration parameter generation network, and the configuration parameter vector is output. In this embodiment, the configuration parameter generation network can adopt the following color matching strategy: when the display attribute is high, a high contrast color scheme (such as a high contrast combination of red-white or orange-black) is preferred; when the display attribute is low, a soft color scheme corresponding to the current emotion category of the interactive object is preferred (such as a blue-white combination when the emotion of the interactive object is calm). The generation method of the configuration parameter vector can be expressed by the following formula (24): in, It can represent the generated configuration parameters, with dimensions of , This represents the total number of configuration parameters. (*) can characterize the network generated by configuring parameters.

[0105] It should be noted that facial parameters, voice parameters, and configuration parameters together constitute the target attribute information. These three types of parameters are all parameterized and generated by the corresponding sub-networks in the preset generation model based on fused features or content semantic features. They are directly regressed in a continuous parameter space, thus enabling fine-grained parameter adjustment based on the emotional state of the interactive object and the display attributes of the content to be displayed.

[0106] S34: Based on the fusion features, the target pose information is obtained by filtering from the preset pose library corresponding to the preset generation model.

[0107] The preset pose library can be understood as a pre-built database used to store multiple candidate poses that can be executed by the virtual object to be interacted with. The preset pose library can include at least one pose, each pose corresponding to a pose feature and a pose parameter. Each pose also carries an emergency adaptability parameter and an emotion label vector: the emergency adaptability parameter is a preset binary parameter used to characterize whether the pose is suitable for high-urgency scenarios or non-urgency scenarios; the emotion label vector is a K-dimensional real number vector, where the k-th component represents the degree of matching between the pose and the k-th emotion category. For example, taking the pose "serious gaze" as an example, its corresponding initial pose feature encodes semantic information related to "serious" and "gaze", its corresponding pose parameter records parameters such as the rotation angle and movement speed of each joint node under this pose, its corresponding emergency adaptability parameter is 1 (indicating suitability for high-urgency scenarios), and the component value of the "serious" related emotion category in its corresponding emotion label vector is relatively high.

[0108] The method of obtaining target pose information by filtering from the preset pose library corresponding to the preset generation model based on fusion features can be as follows: Based on fusion features, an initial number of initial pose features are selected from the preset pose library corresponding to the preset generation model, and the initial pose information corresponding to the initial pose features is obtained; compatibility calculation is performed on the initial pose information to obtain compatibility data, and candidate pose information is selected from the initial number of initial pose information based on the compatibility data; the pose parameters in the candidate pose information are modulated to obtain the target pose information.

[0109] Here, "initial quantity" can be understood as the number of initial poses dynamically determined based on display attributes and initially selected from a preset pose library. "Initial pose features" can be understood as the pose feature representations corresponding to the initial poses retained after initial selection, used to characterize the semantic meaning of the initial pose. "Initial pose information" can be understood as the complete descriptive information corresponding to the initial poses retained after initial selection, including the initial pose features, initial pose parameters, urgency fit parameters, and emotion tag vector. "Compatibility data" can be understood as a numerical value characterizing the compatibility between the initial pose information and the current display attributes; a higher compatibility data indicates that the initial pose is more suitable for use in the current display attribute scenario. "Candidate pose information" can be understood as the initial pose information retained after compatibility selection, which is the object of subsequent pose parameter modulation.

[0110] The method of selecting an initial number of initial pose features from a preset pose library corresponding to a preset generation model based on fusion features and obtaining the initial pose information corresponding to the initial pose features can specifically include: first calculating the similarity between the fusion features and the pose features corresponding to each pose in the preset pose library, then sorting them from high to low similarity, dynamically determining the initial number based on display attributes, selecting the first number of initial poses in the sorted list, using the pose features corresponding to these initial poses as the selected initial pose features, and using the pose information corresponding to these initial poses as the initial pose information.

[0111] Specifically, the fusion features and the pose features corresponding to each pose in the preset pose library are first obtained. L2 normalization is performed on each pose feature, and then the cosine similarity between the fusion features and each normalized pose feature is calculated. The cosine similarity is used as the similarity between the fusion features and the initial pose features. The calculation method for cosine similarity can be expressed by the following formula (25): in, It can characterize the cosine similarity between the fused features and the pose features corresponding to the i-th pose in the preset pose library; It can characterize the L2-normalized initial pose features corresponding to the i-th initial pose.

[0112] After calculating the cosine similarity between the fused features and each initial pose, they are sorted from high to low according to the cosine similarity. The initial number is dynamically determined based on the display attributes, and the first few initial poses in the sorted list are selected as the initial pose information. The determination of the initial number can be expressed by the following formula (26): in, It can represent the initial quantity; It can represent the maximum initial quantity, such as 15; This can represent the minimum initial quantity, which can be 5; It can represent the magnitude of change in the initial quantity, such as taking 10; It can represent display attributes; It can characterize a high display attribute threshold, which can be 0.8; This can characterize a low display attribute threshold, which can be 0.4. The formula indicates that when the display attribute is high, more initial poses are retained to increase selection diversity; when the display attribute is low, fewer initial poses are retained to improve selection efficiency.

[0113] Optionally, in some embodiments, after obtaining the initial posture information, compatibility calculations can be performed on the initial posture information to obtain compatibility data. The compatibility calculation method may specifically include: for each initial posture in the initial posture information, obtaining the emergency adaptability parameter corresponding to that initial posture; then, based on the emergency adaptability parameter and the display attribute, calculating the compatibility between the initial posture and the current display attribute; and using this compatibility as the compatibility data corresponding to that initial posture. The compatibility calculation method can be expressed by the following formula (27): in, It can characterize the compatibility between the i-th initial pose and the current display attribute, that is, the compatibility data corresponding to the initial pose; It can characterize the emergency adaptability parameter corresponding to the i-th initial posture. The value is 1 when the initial posture is a serious posture and 0 when the initial posture is a relaxed posture.

[0114] Optionally, in some embodiments, after obtaining the compatibility data, candidate pose information can be obtained by filtering from the initial number of initial pose information based on the compatibility data. The filtering method may specifically include: for each initial pose in the initial pose information, obtaining the compatibility data corresponding to that initial pose; when the compatibility data value is 0, removing that initial pose from the initial pose information; when the compatibility data value is not 0, retaining that initial pose. The initial pose information retained after filtering is the candidate pose information.

[0115] Optionally, in some embodiments, after obtaining the candidate pose information, the pose parameters in the candidate pose information can be modulated to obtain the target pose information. The modulation method can specifically include the following steps: (1) First, calculate the emotion matching degree between each candidate posture and the emotion of the interactive object in the candidate posture information. The emotion matching degree can be calculated using the following formula (28): in, It can represent the degree of matching between the i-th candidate posture information and the emotion of the interactive object; It can represent the emotion label vector corresponding to the i-th candidate pose information; It can characterize the degree of matching between the candidate pose and the k-th emotion, and is pre-stored in a preset pose library.

[0116] (2) Based on cosine similarity, emotion matching degree, and compatibility data, calculate the comprehensive weight of each candidate pose information. The comprehensive weight can be calculated using the following formula (29): in, It can represent the comprehensive weight of the i-th candidate pose information; It can represent the weight coefficients corresponding to cosine similarity; It can represent the weight coefficient corresponding to the degree of emotional matching; It can represent the weight coefficients corresponding to compatibility.

[0117] (3) After calculating the comprehensive weight of each candidate attitude information, the candidate attitude information with the largest comprehensive weight can be selected as the attitude information to be modulated. The method for determining the attitude information to be modulated can be expressed by the following formula (30): in, It can characterize the attitude information to be modulated, that is, the candidate attitude information with the largest comprehensive weight; It can represent the set of candidate pose information that is retained after compatibility filtering; It can represent the i-th candidate pose information in the candidate pose information set.

[0118] (4) After determining the attitude information to be modulated, obtain the attitude parameters carried in the attitude information to be modulated. The attitude parameters of the attitude information to be modulated are the original parameters of the initial attitude stored in the preset attitude library, which record the original rotation angle and original motion speed of each attitude node under the attitude. At the same time, obtain the interaction activity of the interactive object. The interaction activity can characterize the degree of attention and physical participation of the interactive object to the screen at present or before. It is a dimensionless scalar with a value range of (0,1).

[0119] (5) After obtaining the attitude parameters and interaction activity of the attitude information to be modulated, the attitude parameters of the attitude information to be modulated are modulated based on the display attributes and interaction activity to obtain the modulated attitude parameters. The modulated attitude parameters and the attitude nodes carried in the attitude information to be modulated together constitute the target attitude information. The modulation method can be expressed by the following formula (31): in, It can characterize the modulated attitude parameters; Attitude parameters that can characterize the attitude information to be modulated; It can characterize the interactive activity level of interactive objects; It can be used to characterize the modulation coefficient of interaction activity on attitude parameters, and can be taken as (0.2, 0.8). It can characterize the modulation coefficient of the display attribute on the attitude parameter, and can be taken as (0.3, 1).

[0120] It should be noted that the attitude nodes in the target attitude information are derived from the attitude nodes defined in the attitude information to be modulated in the preset attitude library, and the attitude parameters in the target attitude information are the attitude parameters after the above modulation processing. The purpose of modulation is to increase the amplitude and speed of the virtual object's movements when the interaction activity is high, in response to the user's level of participation; and to increase the amplitude and speed of the virtual object's movements when the display attributes are high, in order to enhance the warning effect of emergency information.

[0121] 104. Generate interactive virtual objects in the target display area based on target attribute information and target posture information, and broadcast and display the content to be displayed in the target display area according to the interactive virtual objects.

[0122] The method of generating interactive virtual objects in the target display area based on target attribute information and target posture information can be specifically described by the following steps S41-S42: S41: Using a preset generation model, the target pose of the virtual object to be interacted with is determined based on the target pose information, so as to drive the virtual object to be interacted with to perform the target pose in the target display area.

[0123] The target pose can be understood as the final physical state of the virtual object to be interacted with in the target display area. This physical state is determined by the pose nodes and pose parameters in the target pose information and is used to control the rotation angle and movement speed of each joint node of the virtual object.

[0124] The method of using a preset generation model to determine the target posture of the virtual object to be interacted with based on the target posture information, so as to drive the virtual object to be interacted with to perform the target posture in the target display area, may specifically include: using a preset generation model to map posture nodes to the joint nodes corresponding to the virtual object to be interacted with; determining the target posture of the virtual object to be interacted with based on the posture parameters corresponding to the posture nodes, and driving the joint nodes of the virtual object to be interacted with to perform the target posture in the target display area.

[0125] In this context, a joint node can be understood as a controllable rotatable bone connection point in the skeletal model of the virtual object to be interacted with, such as head joint nodes, shoulder joint nodes, elbow joint nodes, wrist joint nodes, hip joint nodes, knee joint nodes, etc. Each joint node corresponds to a pose node, which is used to identify the position and name of the joint node.

[0126] The method of mapping pose nodes to the corresponding joint nodes of the virtual object to be interacted with, using a pre-defined generation model, is as follows: First, obtain the pose nodes carried in the target pose information, where each pose node is a joint node identifier defined in the target pose information; then, search for joint nodes that match the pose node identifiers in the skeletal model of the virtual object to be interacted with, establishing a correspondence between pose nodes and joint nodes; finally, associate the pose parameters corresponding to the pose nodes with the matched joint nodes to complete the mapping. It should be noted that the mapping relationship between pose nodes and joint nodes is predefined. The pre-defined generation model stores a correspondence table between pose node identifiers and joint node identifiers, and the mapping is performed directly by searching and matching according to this correspondence table.

[0127] The method of determining the target pose of the virtual object to be interacted with based on the pose parameters corresponding to the pose nodes, and driving the joint nodes of the virtual object to execute the target pose in the target display area, can specifically include: after completing the mapping from pose nodes to joint nodes, obtaining the pose parameters corresponding to each pose node, including the target rotation angle and target motion speed of the pose node; then, based on the target rotation angle and target motion speed corresponding to each joint node, driving each joint node of the virtual object to rotate according to the corresponding pose parameters, so that the virtual object to be interacted with executes the target pose in the target display area. The rotational motion of the joint nodes can be implemented through a skeletal animation driving engine, which calculates the rotation interpolation of the joint nodes frame by frame based on the target rotation angle and target motion speed of the joint nodes, generating smooth skeletal animation.

[0128] It should be noted that the execution effect of the target posture is affected by the modulation processing in step 103. When the interaction activity is high or the display attribute is high, the modulated posture parameters are increased based on the basic posture parameters, the rotation angle and movement speed of the joint nodes are increased accordingly, and the movement amplitude and movement speed of the target posture are more significant.

[0129] S42: Using a preset generation model, the target facial action of the virtual object to be interacted with is determined based on the target attribute information, so as to drive the virtual object to be interacted with to perform the target facial action in the target display area.

[0130] Among them, the target facial motion can be understood as the comprehensive performance of the facial expression state, mouth movement state and voice broadcast state of the virtual object to be interacted with in the target display area, which is determined by the facial parameters, voice parameters and configuration parameters in the target attribute information.

[0131] The method involves using a preset generation model to determine the target facial movements of the virtual object to be interacted with based on target attribute information, thereby driving the virtual object to perform the target facial movements in the target display area. Specifically, this may include: using a preset generation model to determine the facial expression corresponding to the virtual object to be interacted with based on facial parameters; generating the voice to be played and mouth movements corresponding to the virtual object to be interacted with based on the content to be displayed and voice parameters; generating the display color corresponding to the virtual object to be interacted with based on configuration parameters; obtaining the target facial movements of the virtual object to be interacted with based on the facial expressions, the voice to be played, the mouth movements, and the display color; and driving the virtual object to be interacted with to perform the target facial movements in the target display area.

[0132] The method of determining the facial expression corresponding to the virtual object to be interacted with based on a preset generation model and facial parameters can specifically include: acquiring the facial parameters output in step S33, including parameters such as the degree of mouth opening, the angle of mouth corners, the height of eyebrows, and the degree of eye opening and closing; then mapping the facial parameters onto the facial deformation units of the virtual object to be interacted with, where the facial deformation units are controllable deformation areas in the virtual object's facial model, such as mouth deformation units, eyebrow deformation units, and eye deformation units; finally, driving the corresponding facial deformation units to deform according to each parameter value in the facial parameters, so that the virtual object to be interacted with presents a facial expression corresponding to the facial parameters. For example, when the angle of mouth corners is positive, the mouth deformation unit is driven to deform upwards, presenting a smiling expression; when the eyebrow height parameter increases, the eyebrow deformation unit is driven to deform upwards, presenting a surprised expression.

[0133] The method of generating the speech to be played and mouth movements corresponding to the virtual object to be interacted with, based on the content to be displayed and the speech parameters, can specifically include: first, obtaining the content to be displayed determined in step 102 and the speech parameters output in step S33, including speech rate parameters, pitch fundamental frequency parameters, and volume parameters; then, inputting the text information of the content to be displayed and the speech parameters into the speech synthesis module, and using the speech synthesis module to synthesize the content to be displayed according to the speech rate, pitch fundamental frequency, and volume in the speech parameters to obtain the speech to be played; during the synthesis of the speech to be played, mouth movement parameters of the virtual object to be interacted with are generated synchronously according to the rhythm and prosody information of the speech to be played, and the mouth movement parameters are used to control the opening and closing degree and rhythm of the virtual object's mouth; finally, mapping the mouth movement parameters to the mouth deformation unit of the virtual object to be interacted with, driving the mouth deformation unit to open and close according to the rhythm and prosody of the speech to be played, so that the mouth movements of the virtual object are aligned with the speech to be played, and lip-syncing is achieved.

[0134] The method of generating the display color corresponding to the virtual object to be interacted with based on configuration parameters can specifically include: obtaining the configuration parameters output in step S33, which include the RGBA values ​​of parameters such as the color of the virtual character's clothing, background color, text color, and highlight color; then, applying the RGBA values ​​of each color in the configuration parameters to the clothing material of the virtual object to be interacted with, the background of the target display area, the text rendering of the content to be displayed, and the rendering of the highlighted marks, so that the display color of the virtual object to be interacted with and the broadcast interface is consistent with the configuration parameters. Since the configuration parameters are generated based on fusion features and display attributes in step S33, the presentation effect of the display color is related to the display attributes: when the display attributes are high, the configuration parameters correspond to a high-contrast color scheme (such as a red-white or orange-black combination); when the display attributes are low, the configuration parameters correspond to a soft color scheme that matches the current emotion category of the interactive object (such as a blue-white combination for the "calm" emotion).

[0135] The method of obtaining the target facial movement of the virtual object to be interacted with, based on facial expressions, the voice to be played, mouth movements, and display colors, can specifically include: combining the aforementioned determined facial expressions, the voice to be played, mouth movements, and display colors to form the complete facial expression state of the virtual object to be interacted with when playing the content to be displayed in the target display area; this complete facial expression state is the target facial movement. Specifically, facial expressions and mouth movements work together to make the virtual object present a facial expression that matches the voice content and emotional state while playing the voice; display colors affect the appearance and material of the virtual object and the background of the playback interface, enhancing the overall performance effect.

[0136] Specifically, the method of driving the virtual object to be interacted with to perform the target facial action in the target display area may include: inputting the facial deformation parameters, mouth action parameters, voice data and display color parameters corresponding to the target facial action into the virtual image rendering engine in the preset generation model; driving the facial deformation unit of the virtual object to be interacted with to deform according to the facial deformation parameters; driving the mouth deformation unit to open and close according to the mouth action parameters; playing the voice to be broadcast; and rendering the appearance material of the virtual object and the background of the target display area according to the display color parameters, so that the virtual object to be interacted with performs the target facial action in the target display area.

[0137] It should be noted that the target pose determined in step S41 and the target facial movement determined in step S42 are executed synchronously in the time dimension. That is, while the virtual object to be interacted with is performing the target pose in the target display area, it is also performing the target facial movement. Together, they constitute the complete performance state of the virtual object when broadcasting the content to be displayed. The target pose controls the virtual object's body movements, while the target facial movement controls the virtual object's facial expression, voice broadcast, and display color. Their synergy ensures that the virtual object's broadcast performance matches the emotional state of the interactive object and the display attributes of the content to be displayed.

[0138] It should also be noted that after the virtual object to be interacted with is generated, the content to be displayed can be broadcast and shown on the target display area based on the virtual object. The broadcast and display process includes: rendering the target pose and facial movements of the virtual object to be interacted with in the target display area, broadcasting the content to be displayed in voice form, and displaying the text information of the content to be displayed in the target display area. During the broadcast and display process, other functional areas of the display terminal (such as the content display area, information overview area, and to-do list area) can simultaneously display their corresponding information content.

[0139] In this embodiment of the application, while the virtual object to be interacted with is broadcasting and displaying the content to be displayed on the target display area, the interaction data (such as action change data) of the interactive object can be acquired simultaneously to determine the real-time interactive activity of the interactive object. The posture parameters of the virtual object to be interacted with can be updated according to the real-time interactive activity, so that the virtual object to be interacted with can dynamically adjust its own action performance according to the real-time participation of the interactive object during the broadcasting process, forming a real-time interactive feedback loop of interactive object state change - virtual object performance response.

[0140] Specific implementation methods may include: extracting motion change data of the interactive object from at least one target video frame of the video stream; calculating the gaze attention of the interactive object based on the motion change data; determining the interaction activity parameter of the interactive object based on the motion change data and gaze attention; and modulating the posture parameters in the candidate posture information based on the interaction activity parameter to obtain updated posture information.

[0141] In this context, the target video frame can be understood as a video frame of the interactive object captured in real time during the process of the virtual object playing and displaying the content to be displayed. The target video frame can be a subsequent frame in the video stream after the video frame used in the initial recognition stage, that is, a video frame captured in the real-time interaction stage, used to reflect the real-time state changes of the interactive object during the virtual object playing process.

[0142] Among these, motion change data can be understood as data extracted from target video frames to reflect the limb movements of the interactive object, including displacement data of key points of the interactive object's body between adjacent target video frames. Gaze attention can be understood as a value characterizing the consistency between the interactive object's gaze direction and the normal direction of the display terminal screen; higher gaze attention indicates a more focused interactive object on the screen. Interaction activity parameters can be understood as values ​​characterizing the interactive object's real-time participation and attention during the broadcast; higher interaction activity parameters indicate a higher level of attention and physical involvement from the interactive object. Updated posture information can be understood as new posture information obtained by modulating the current posture parameters of the virtual object to be interacted with based on the real-time interaction activity parameters, used to dynamically adjust the virtual object's motion performance during the broadcast.

[0143] The method of extracting motion change data of interactive objects from at least one target video frame of a video stream can specifically include: first, acquiring a real-time captured sequence of target video frames, which includes at least two consecutive target video frames; then, using a limb motion feature extraction network to extract the human body key point coordinate sequence of the interactive object from each target video frame; next, calculating the displacement of the same human body key point between two adjacent target video frames to obtain the displacement data of each human body key point; finally, using the displacement data of each human body key point as motion change data.

[0144] The method of calculating the gaze attention of an interactive object based on its motion change data can specifically include: first, determining the head orientation of the interactive object based on the displacement changes of key head points in the motion change data; then, estimating the gaze direction of the interactive object based on its head orientation; and finally, calculating the cosine similarity between the estimated gaze direction and the normal direction of the display terminal screen to obtain the gaze attention. The gaze attention is a dimensionless scalar with a value range of [0,1]. When the interactive object is looking directly at the screen, the gaze attention approaches 1; when the interactive object's gaze is completely off-screen, the gaze attention approaches 0.

[0145] The method for determining the interaction activity parameter of an interactive object based on motion change data and gaze attention can specifically include: first, calculating the average displacement of all human keypoints between adjacent frames based on the displacement of each keypoint in the motion change data to obtain the limb movement amplitude; then, obtaining preset weighted coefficients for limb movement amplitude and gaze attention, multiplying the limb movement amplitude by the weighted coefficients, multiplying the gaze attention by the weighted coefficients, and then adding the two products; finally, inputting the sum into a Sigmoid activation function for mapping to obtain the interaction activity parameter. The interaction activity parameter is a dimensionless scalar with a value range of (0,1).

[0146] The method of modulating the posture parameters in the candidate posture information based on the interaction activity parameter to obtain the updated posture information can specifically include: first, obtaining the posture parameters in the currently executing target posture information; then, obtaining a preset interaction activity modulation coefficient; next, multiplying the interaction activity parameter by the interaction activity modulation coefficient and adding 1 to obtain a modulation coefficient; finally, multiplying the current posture parameter by the modulation coefficient to obtain the updated posture parameter. The updated posture parameter and the current posture node together constitute the updated posture information. After obtaining the updated posture information, it can be input into the virtual image rendering engine to replace or smoothly update the current target posture information, driving the virtual object to be interacted with to adjust its action performance according to the updated posture information.

[0147] It should be noted that the interaction activity parameter in this implementation is the same type of parameter used when initially determining the target posture information; both are used to characterize the degree of attention and physical engagement of the interactive object to the screen. The difference lies in the calculation method: the interaction activity parameter used when initially determining the target posture information is calculated based on video frames within the initial time window during the initial recognition phase, reflecting the interactive object's participation state before the virtual object begins playback; the interaction activity parameter in this implementation is calculated based on real-time acquired target video frames during playback, reflecting the interactive object's real-time participation state during the virtual object playback process. By updating the posture parameters using the real-time interaction activity parameter, the virtual object's actions can be dynamically adjusted according to the real-time participation of the interactive object, improving the real-time nature and naturalness of the interaction.

[0148] Furthermore, in the embodiments of this application, during the process of the virtual object to be interacted with broadcasting and displaying the content to be displayed on the target display area, such as Figure 3 As shown, the display area of ​​the display terminal can be divided into multiple functional zones according to a preset layout, with each functional zone allocated an independent frame buffer storage area. The target display area is the virtual avatar display area among the multiple functional zones, used to display the animation of the virtual object to be interacted with. In addition to the target display area, the multiple functional zones may also include a content display area, an information overview area, and a to-do list area. The content display area displays the text information of the content to be displayed, the information overview area displays time and calendar information, and the to-do list area displays a list of to-do items. During the broadcast, the refresh priority of each functional zone can be dynamically determined according to the display attributes of the content to be displayed, and the refresh frame rate of each functional zone during the broadcast and display of the content to be displayed can be controlled according to the refresh priority.

[0149] Specifically, each functional area has a preset basic refresh priority and an emergency response coefficient. The content display area has the highest emergency response coefficient, followed by the target display area, while the information overview area and to-do list area have lower coefficients. The refresh priority weight of each functional area is determined by the basic refresh priority, the emergency response coefficient, display attributes, and the emotional intensity of the interactive object calculated in step 101. Specifically, the emotional intensity of the interactive object can be used as a moderating factor in the calculation of the refresh priority weight of each functional area: when the emotional intensity of the interactive object is high, it indicates that the interactive object is currently in a relatively positive or strong emotional state. In this case, the refresh priority weight of the target display area can be appropriately increased to enhance the smoothness and subtlety of the virtual object's actions, thereby better responding to the user's emotional state; when the emotional intensity of the interactive object is low or negative, it indicates that the interactive object is currently in a relatively negative or calm emotional state. In this case, the basic allocation of the refresh priority weight of each functional area can be maintained to ensure the stability of the overall display effect.

[0150] After incorporating the emotional intensity of interactive objects into the refresh priority weight calculation, the refresh priority weight of each functional area can be determined according to the following rules: Based on the basic refresh priority of each functional area, the product of display attributes and the emergency response coefficient is used as the emergency adjustment item, and the product of the emotional intensity of the interactive object and the emotional response coefficient is used as the emotional adjustment item. The sum of the basic refresh priority, the emergency adjustment item, and the emotional adjustment item is used as the refresh priority weight of that functional area. Specifically, the emotional response coefficient of the target display area can be set to be greater than that of other functional areas, so that changes in the emotional intensity of the interactive object have the most significant impact on the refresh priority of the target display area.

[0151] Based on the above rules, when the display attributes are high, the emergency adjustment items for the content display area and the target display area increase, and the refresh priority weight increases accordingly, ensuring the timeliness of key information refresh and the smoothness of virtual object actions in emergency scenarios. When the display attributes are low, the emergency adjustment items for each functional area decrease, and the refresh priority weight tends to the sum of the base value and the emotion adjustment item. At this time, the target display area can still obtain relatively sufficient refresh resources when the emotional intensity of the interactive object is high, so as to ensure the smoothness of the virtual character animation and the ability to respond to emotions. By dynamically scheduling the refresh priority of each functional area, under the condition that the total system refresh resources are limited, the effect of prioritizing the refresh of key areas in emergency scenarios and prioritizing the refresh of the virtual character display area in non-emergency scenarios, and adaptively adjusting according to the user's emotional state can be achieved, improving the utilization rate of display resources and the efficiency of information transmission. Furthermore, the above dynamic scheduling of refresh priorities is continuously executed throughout the entire process of the virtual object to be interacted with broadcasting and displaying the content to be displayed. When the display attributes change, the refresh priority of each functional area is updated accordingly.

[0152] As described above, this embodiment of the application, after acquiring a video stream for the interactive object and identifying at least one video frame in the video stream to obtain the current emotion category of the interactive object, can determine the content to be displayed corresponding to the target display area, and perform content recognition on the content to be displayed to obtain the display attributes corresponding to the content to be displayed. Then, based on the current emotion category of the interactive object and the display attributes of the content to be displayed, the target attribute information and target posture information corresponding to the virtual object to be interacted with are determined. Subsequently, based on the target attribute information and target posture information, the virtual object to be interacted with is generated in the target display area, and the content to be displayed is broadcast and displayed on the target display area according to the virtual object to be interacted with. Since this scheme can dynamically determine the target attribute information and target posture information of the virtual object according to the current emotion category of the interactive object and the display attributes of the content to be displayed, the performance of the virtual object can match both the user's emotional state and the urgency of the broadcast content, thereby improving the accuracy of information transmission, enabling users to intuitively understand the information represented by the data, and thus ensuring user experience.

[0153] To better implement the above methods, this application also provides a display interaction system that can be integrated into an electronic device, such as a server or terminal. The terminal may include a tablet computer, a laptop computer, and / or a personal computer.

[0154] For example, such as Figure 4 As shown, the display interaction system may include an information acquisition unit 301, a content recognition unit 302, an information determination unit 303, and a display interaction unit 304, as follows: (1) Information acquisition unit 301; The information acquisition unit 301 is used to acquire a video stream for the interactive object and identify at least one video frame in the video stream to obtain the current emotion category of the interactive object.

[0155] (2) Content recognition unit 302; The content recognition unit 302 is used to determine the content to be displayed corresponding to the target display area, and to perform content recognition on the content to be displayed in order to obtain the display attributes corresponding to the content to be displayed.

[0156] (3) Information determination unit 303; The information determination unit 303 is used to determine the target attribute information and target posture information corresponding to the virtual object to be interacted with based on the current emotion category of the interactive object and the display attributes of the content to be displayed.

[0157] (4) Display interaction unit 304; The display interaction unit 304 is used to generate the virtual object to be interacted in the target display area based on the target attribute information and the target posture information, and to broadcast and display the content to be displayed on the target display area according to the virtual object to be interacted.

[0158] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0159] This application also provides an electronic device, such as... Figure 5 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically: The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 402, and calls data stored in the memory 402, to perform various functions and process data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0160] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0161] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0162] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0163] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions.

[0164] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0165] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the display interaction methods provided in embodiments of this application.

[0166] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0167] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0168] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the display interaction methods provided in the embodiments of this application, the beneficial effects that any of the display interaction methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0169] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various alternative implementations of the above-described display interaction aspects.

[0170] The foregoing has provided a detailed description of a display interaction method and related devices provided in the embodiments of this application. These related devices may include display interaction systems, electronic devices, computer program products, and computer-readable storage media. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are merely for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there may be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A display interaction method, characterized in that, The target display area applied to the display terminal includes: Acquire a video stream for the interactive object, and identify at least one video frame in the video stream to obtain the current emotion category of the interactive object; Determine the content to be displayed corresponding to the target display area, and perform content recognition on the content to be displayed to obtain the display attributes corresponding to the content to be displayed; Based on the current emotion category of the interactive object and the display attributes of the content to be displayed, determine the target attribute information and target posture information corresponding to the virtual object to be interacted with; Based on the target attribute information and the target posture information, the virtual object to be interacted with is generated in the target display area, and the content to be displayed is broadcast and displayed on the target display area according to the virtual object to be interacted with.

2. The display interaction method according to claim 1, characterized in that, The step of determining the target attribute information and target posture information corresponding to the virtual object to be interacted with based on the current emotion category of the interactive object and the display attributes of the content to be displayed includes: Semantic features are extracted from the current emotion category and the display attribute to obtain the emotion features corresponding to the current emotion category and the display attribute features corresponding to the display attribute. The emotional features and the display attribute features are weighted and fused to obtain the fused features; The target attribute information is obtained by parametric generation based on the fusion features using a preset generation model; Based on the fusion features, the target pose information is obtained by filtering from the preset pose library corresponding to the preset generation model.

3. The display interaction method according to claim 2, characterized in that, Based on the fusion features, the target pose information is obtained by filtering from the preset pose library corresponding to the preset generation model, including: Based on the fusion features, an initial number of initial pose features are selected from the preset pose library corresponding to the preset generation model, and the initial pose information corresponding to the initial pose features is obtained. Compatibility calculations are performed on the initial attitude information to obtain compatibility data, and candidate attitude information is obtained by filtering from the initial number of initial attitude information based on the compatibility data; The attitude parameters in the candidate attitude information are modulated to obtain the target attitude information.

4. The display interaction method according to claim 2, characterized in that, The step of generating the interactive virtual object in the target display area based on the target attribute information and the target posture information includes: Using the preset generation model, the target pose of the virtual object to be interacted with is determined based on the target pose information, so as to drive the virtual object to be interacted with to perform the target pose in the target display area; Using the preset generation model, the target facial action of the virtual object to be interacted with is determined based on the target attribute information, so as to drive the virtual object to be interacted with to perform the target facial action in the target display area.

5. The display interaction method according to claim 4, characterized in that, The target pose information includes pose nodes and pose parameters. The step of using the preset generation model to determine the target pose of the virtual object to be interacted with based on the target pose information, so as to drive the virtual object to be interacted with to execute the target pose in the target display area, includes: Using the preset generation model, the pose nodes are mapped to the joint nodes corresponding to the virtual object to be interacted with; The target pose of the virtual object to be interacted with is determined based on the pose parameters corresponding to the pose node, and the joint nodes of the virtual object to be interacted with are driven to perform the target pose in the target display area.

6. The display interaction method according to claim 4, characterized in that, The target attribute information includes facial parameters, voice parameters, and configuration parameters. The step of using the preset generation model to determine the target facial action of the virtual object to be interacted with based on the target attribute information, so as to drive the virtual object to be interacted with to perform the target facial action in the target display area, includes: Using the preset generation model, the facial expression corresponding to the virtual object to be interacted with is determined based on the facial parameters; Based on the content to be displayed and the voice parameters, generate the voice to be played and the mouth movements corresponding to the virtual object to be interacted with; The display color corresponding to the virtual object to be interacted with is generated based on the configuration parameters; The target facial movement of the virtual object to be interacted with is obtained based on the facial expression, the voice to be broadcast, the mouth movement, and the display color. The virtual object to be interacted with is instructed to perform the target facial action in the target display area.

7. The display interaction method according to claim 1, characterized in that, The step of identifying at least one video frame in the video stream to obtain the current emotion category of the interactive object includes: Extract multimodal data of the interactive object from at least one video frame of the video stream; Feature extraction is performed on each modal data in the multimodal data to obtain the modal features of each modal data; A feature extraction model is used to determine the modal features of each modal data and the attention weights corresponding to the modal features, so as to obtain the weighted modal features; The weighted modal features are fused to obtain the emotion distribution data of the interactive object, and the current emotion category of the interactive object is determined based on the emotion distribution data.

8. The display interaction method according to claim 1, characterized in that, The step of determining the content to be displayed corresponding to the target display area and performing content recognition on the content to be displayed to obtain the display attributes corresponding to the content to be displayed includes: The content to be displayed corresponding to the target display area is obtained from the display terminal, the semantics of the content to be displayed is extracted to obtain the content semantic features, and the urgency of the content semantic features corresponding to the content to be displayed is classified to obtain the content urgency corresponding to the content semantic features. Acquire multi-source environmental sensor data, and perform urgency-normalized mapping on each environmental sensor data to obtain the environmental sensor urgency level. Obtain external early warning data corresponding to a preset platform, and map the external early warning data to the urgency of external information; The urgency of the content, the urgency of the environmental sensors, and the urgency of the external information are weighted and fused according to preset weights to obtain the display attributes corresponding to the content to be displayed.

9. The display interaction method according to claim 3, characterized in that, The method further includes: Extract the motion change data of the interactive object from at least one target video frame of the video stream; The gaze attention level of the interactive object is calculated based on the action change data of the interactive object. Based on the action change data and the gaze attention, the interaction activity parameter of the interactive object is determined; The posture parameters in the candidate posture information are modulated based on the interaction activity parameters to obtain updated posture information.

10. A display interaction system, characterized in that, The target display area applied to the display terminal includes: An information acquisition unit is used to acquire a video stream for an interactive object and to identify at least one video frame in the video stream to obtain the current emotion category of the interactive object. A content recognition unit is used to determine the content to be displayed corresponding to the target display area, and to perform content recognition on the content to be displayed in order to obtain the display attributes corresponding to the content to be displayed; The information determination unit is used to determine the target attribute information and target posture information corresponding to the virtual object to be interacted with, based on the current emotion category of the interactive object and the display attributes of the content to be displayed. The display interaction unit is used to generate the virtual object to be interacted with in the target display area based on the target attribute information and the target posture information, and to broadcast and display the content to be displayed in the target display area according to the virtual object to be interacted with.

Images