Digital human vision line control method and system combined with group behavior recognition

Abstract

The application discloses a digital human visual line control method and system combined with group behavior recognition, and relates to the technical field of digital human control. The method comprises the following steps: in response to a multi-path video stream input, identifying participants and extracting a behavior parameter set; calculating a deviation angle based on a head pitch angle time sequence, and obtaining a social attention weight by combining a shoulder orientation angle; calculating an interaction activity based on a shoulder key point displacement trajectory and a direction change sequence; taking the social attention weight and the interaction activity as multiplicative constraints to correct an audio energy value, and generating a dynamic visual line target sequence; controlling the visual line focus to move between targets by interpolation to obtain a smooth path, and adjusting a starting slope by combining a social attention weight ratio of adjacent targets to generate a visual line movement trajectory. The application combines head posture, shoulder orientation, limb movement and audio energy multi-dimensional behavior characteristics, solves the problem that traditional methods only rely on audio, leading to harsh visual line switching, and improves the naturalness of digital human interaction and user reach rate.

Description

Technical Field

[0001] This invention relates to the field of digital human control technology, specifically to a digital human gaze control method and system that combines group behavior recognition. Background Technology

[0002] Digital humans are computer-generated virtual characters that exist in the digital world and possess human-like form and interactive capabilities. In multi-person video conferencing scenarios, AI virtual assistants, as a typical application of digital humans, need to simulate natural interpersonal interaction behaviors through gaze control, such as switching gaze focus among multiple participants to indicate the object of attention and participation status.

[0003] Traditional methods primarily rely on audio energy to detect the current speaker and drive the digital human's gaze to simply follow that speaker's movement—that is, focusing on whoever speaks loudest. This method is simple to implement, computationally inexpensive, and generally meets the needs in single-person speaking scenarios. However, depending solely on audio energy to follow the speaker easily leads to frequent gaze jumps in interruption scenarios, an inability to identify a single gaze target when multiple people are speaking simultaneously, and a lack of clear gaze direction during silence periods, resulting in abrupt and jarring gaze transitions. Furthermore, this method ignores nonverbal social cues such as head posture, shoulder orientation, and body language. Therefore, audio-driven gaze control alone struggles to simulate natural social gaze behavior, reducing the naturalness of interaction between the digital human and the user, and lowering user reach. Summary of the Invention

[0004] This invention addresses the technical problems in existing technologies that rely solely on audio energy to drive gaze, neglecting non-verbal social cues such as head posture and body movements, resulting in stiff gaze switching and low naturalness of interaction. It provides a digital human gaze control method and system that combines group behavior recognition.

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: In a first aspect, the present invention provides a digital human gaze control method incorporating group behavior recognition, comprising: Responding to multiple video stream inputs, the system identifies multiple participants in the video conferencing scenario where the digital human is located as candidate interaction objects and extracts the behavioral parameter set of each participant. Based on the time sequence of each participant's head pitch angle, the deviation angle between the corresponding participant's gaze direction and the digital human's line of sight is calculated, and the social attention weight is obtained based on the deviation angle and the shoulder orientation angle. Based on a continuous sequence of video frame images, combined with the temporal sequence of coordinates of key points on both sides of the shoulder, the corresponding displacement trajectory and direction change sequence of the shoulder key points are generated, and the interaction activity is calculated by combining the displacement amplitude and the number of direction changes. Using the social attention weight and the interaction activity as multiplicative constraints, the instantaneous audio energy value of the corresponding participant is corrected to obtain the corrected energy value. All participants are then sorted from high to low according to the corrected energy value to generate a dynamic gaze target sequence. The digital human's gaze focus is controlled to interpolate and move between targets in the dynamic gaze target sequence to obtain a smooth path between the current gaze focus and the next gaze target. The initial slope of the smooth path is adjusted by combining the social attention weight ratio of adjacent targets to generate a gaze movement trajectory.

[0006] Secondly, this invention provides a digital human gaze control system that incorporates group behavior recognition, comprising: The identification and data acquisition module is used to respond to multiple video stream inputs, identify multiple participants in the video conference scene where the digital human is located as candidate interaction objects, and extract the behavioral parameter set of each participant; The attention weight calculation module is used to calculate the deviation angle between the gaze direction of each participant and the gaze direction of the digital human based on the head pitch angle time sequence of each participant, and to obtain the social attention weight based on the deviation angle and the shoulder orientation angle. The interaction activity calculation module is used to generate the corresponding shoulder key point displacement trajectory and direction change sequence based on the continuous video frame image sequence and the time sequence of the coordinates of key points on both sides of the shoulder, and to calculate the interaction activity by combining the displacement amplitude and the number of direction changes. The gaze target sequence generation module is used to correct the instantaneous audio energy value of the corresponding participant by using the social attention weight and the interaction activity as multiplicative constraints, to obtain the corrected energy value, and to sort all participants from high to low according to the corrected energy value to generate a dynamic gaze target sequence. The gaze movement trajectory generation module is used to control the digital human's gaze focus to move between targets in the dynamic gaze target sequence through interpolation, obtain a smooth path between the current gaze focus and the next gaze target, and adjust the initial slope of the smooth path by combining the social attention weight ratio of adjacent targets to generate the gaze movement trajectory.

[0007] The beneficial effects of this invention are: Compared to existing technologies, this invention first integrates multiple video and audio streams to comprehensively perceive the interaction status of participants from four dimensions: head posture, shoulder orientation, body movements, and voice energy, overcoming the limitations of single audio-driven approaches. Secondly, it calculates social attention weights by weighted fusion of head pitch angle deviation angle and shoulder orientation angle, enabling the digital human to accurately identify participants who are genuinely paying attention and avoid focusing on distracting targets. Thirdly, it quantifies interaction activity based on the displacement trajectory and directional change frequency of key shoulder points, incorporating the social intent of body language into gaze decision-making, enhancing the digital human's ability to perceive proactive interaction intentions. Finally, it uses social attention weights and interaction activity as multiplicative constraints to correct audio energy, generating a dynamically ordered sequence of gaze targets, and employs Bézier curve interpolation and slope adjustment to generate smooth gaze movement trajectories, ensuring that gaze switching conforms to the physiological characteristics of real human interaction. This invention improves the naturalness of gaze interaction and user immersion in multi-person meeting scenarios. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating the digital human gaze control method combining group behavior recognition provided by the present invention. Figure 2 This is a schematic diagram of the structure of the digital human gaze control system that combines group behavior recognition provided by the present invention.

[0009] In the attached diagram, the components represented by each number are as follows: The module includes a recognition and data acquisition module 11, an attention weight calculation module 12, an interaction activity calculation module 13, a gaze target sequence generation module 14, and a gaze movement trajectory generation module 15. Detailed Implementation

[0010] Example 1, as Figure 1 As shown, this embodiment of the invention provides a digital human gaze control method that combines group behavior recognition, including: S10: Responds to multiple video stream inputs, identifies multiple participants in the video conferencing scene where the digital human is located as candidate interaction objects, and extracts the behavioral parameter set of each participant; The behavioral parameter set includes at least the head pitch angle time sequence, the shoulder orientation angle time sequence, and the coordinate time sequence of key points on both sides of the shoulder.

[0011] First, responding to multiple video stream inputs, the system identifies multiple participants in the video conference scenario as candidate interaction targets. In multi-person video conferencing scenarios, the digital human appears on the conference screen as an AI virtual assistant, requiring gaze control to simulate natural interpersonal interaction behavior. Multiple video stream inputs refer to video data streams collected from multiple participants' terminals, with each video stream corresponding to a participant's camera view. By performing real-time human detection and facial recognition on each video stream, participants in the frame can be identified and used as candidate interaction targets—potential targets on which the digital human's gaze might be directed.

[0012] Secondly, extract the behavioral parameter set for each participant. This behavioral parameter set includes at least the head pitch angle time sequence, the shoulder orientation angle time sequence, and the coordinate time sequence of key points on both sides of the shoulder.

[0013] The head pitch angle time series is obtained by calculating the pose of key head points in consecutive video frames. Specifically, facial feature points, such as the positions of eyes, nose, and corners of the mouth, are extracted from each frame. The rotation matrix of the head coordinate system relative to the camera coordinate system is solved using a perspective n-point projection method. The pitch angle is then extracted and arranged in chronological order to form the head pitch angle time series. This head pitch angle time series reflects the changing patterns of the participant's head nodding movements and is an important basis for determining whether they are paying attention to the digital human.

[0014] The shoulder orientation angle time series is obtained by locating the two-dimensional coordinates of the left and right shoulder keypoints and calculating the angle between the line connecting the two shoulders and the horizontal plane. Specifically, for each frame of the image, human pose estimation is performed, the pixel coordinates of the left and right shoulder keypoints are output, and the angle between the line connecting the left and right shoulder keypoints and the horizontal line is calculated. This angle reflects the degree of left and right twisting of the participant's body. These are arranged in chronological order to form the shoulder orientation angle time series. This shoulder orientation angle time series is used to determine whether the participant's body is facing the digital human.

[0015] The temporal sequence of key point coordinates on both sides of the shoulders is obtained by reasoning through each frame of the image, outputting the pixel coordinates of the left and right shoulder key points, and arranging them in chronological order. Specifically, for example, using a deep learning-based human key point detection model, each frame of the image is reasoned through frame by frame, outputting the horizontal and vertical coordinates of the left and right shoulder key points in the image coordinate system, and arranging the coordinates in chronological order to form the temporal sequence of key point coordinates on both sides of the shoulders. This temporal sequence of key point coordinates on both sides of the shoulders is used to subsequently calculate the displacement trajectory and directional changes of the shoulder center point, thereby quantifying the participant's limb activity level.

[0016] By extracting the aforementioned set of behavioral parameters, the digital human can perceive the head posture, body orientation, and limb movements of each participant from a visual perspective, providing a data foundation for subsequent calculation of social attention weights and evaluation of interaction activity.

[0017] S20: Based on the head pitch angle time sequence of each participant, calculate the deviation angle between the corresponding participant's gaze direction and the digital human's gaze direction, and obtain the social attention weight based on the deviation angle and the shoulder orientation angle; Secondly, based on the time sequence of each participant's head pitch angle, the deviation angle between the participant's gaze direction and the digital human's line of sight is calculated. In a multi-person video conferencing scenario, the digital human is usually located in a fixed position in the center of the screen, with its line of sight pointing directly in front of the screen. The direction in which a participant looks at the digital human is determined by their head pitch angle. When a participant is looking directly at the screen, the head pitch angle is close to zero degrees; when a participant looks down at their phone or up at something else, the head pitch angle changes significantly. The deviation angle is the absolute value of the angle between the participant's gaze direction and the digital human's line of sight. The smaller the deviation angle, the more focused the participant is on the digital human; the larger the deviation angle, the more distracted the participant's attention.

[0018] Social attention weights are derived based on the deviation angle and shoulder orientation angle. Social attention weight is a comprehensive quantitative indicator used to assess the degree of attention participants give to the digital avatar. The calculation of this social attention weight integrates information from two dimensions: head posture reflects the participant's instantaneous gaze direction, and body orientation reflects the participant's overall willingness to interact. When a participant is both looking at the digital avatar and facing the screen, their social attention weight is higher; when a participant is looking down or turned to the side, their social attention weight is lower.

[0019] Specifically, the calculation of social attention weight comprehensively considers multiple factors, including instantaneous deviation angle, the drastic change in deviation angle, shoulder orientation angle, and shoulder orientation stability, and obtains the final weight value through weighted fusion. The higher the social attention weight value, the greater the participant's attention to the digital human, and the higher its priority should be in the subsequent ranking of gaze targets.

[0020] Specifically, based on the time sequence of each participant's head pitch angle, the deviation angle between the corresponding participant's gaze direction and the digital human's line of sight is calculated, and social attention weights are obtained based on the deviation angle and shoulder orientation angle, including: Extract the current head pitch angle from the head pitch angle time sequence, calculate the absolute value of the angle between the head pitch angle and the direction of the digital human's line of sight, and obtain the instantaneous deviation angle; The head pitch angles of the N consecutive moments preceding the current moment are extracted from the head pitch angle time sequence. The deviation angle change severity is calculated by combining the pitch angle difference between adjacent moments, where N is a positive integer determined based on the head motion response time and the video frame rate. Obtain the shoulder orientation angle at the current moment, and extract the shoulder orientation angles of the M consecutive moments before the current moment from the shoulder orientation angle time sequence to calculate the shoulder orientation stability, where M is a positive integer determined based on the body rotation response time and the video frame rate; Based on the instantaneous deviation angle and the drastic change of the deviation angle, a gaze offset factor is calculated; Based on the shoulder orientation angle and the shoulder orientation stability, calculate the body attention factor; The social attention weight is obtained by calculating the weighted sum of the reciprocal of the gaze shift factor and the body attention factor.

[0021] First, the head pitch angle at the current moment is extracted from the head pitch angle time series. The absolute value of the angle between the head pitch angle and the digital human's gaze direction is calculated to obtain the instantaneous deviation angle. The digital human's gaze direction usually points directly in front of the screen, corresponding to a pitch angle of zero degrees. The instantaneous deviation angle is equal to the absolute value of the difference between the current head pitch angle and the digital human's gaze direction. The smaller this value, the closer the participant's current gaze direction is to the digital human, and the higher their level of attention.

[0022] Secondly, the head pitch angles of the N consecutive moments preceding the current moment are extracted from the head pitch angle time series. Combined with the pitch angle differences between adjacent moments, the drastic change in the deviation angle is calculated. N is a positive integer determined based on the head motion response time and the video frame rate. The head motion response time is determined based on the rapid response characteristics of the human neck muscle groups to stimuli and the relatively small moment of inertia of head rotation, and is typically taken as 0.2 seconds. The video frame rate is determined based on the real-time and smoothness requirements of video conferencing scenarios and the capabilities of common camera hardware, and is typically taken as 30 frames per second. Optionally, N is equal to the head motion response time multiplied by the video frame rate, and the result is rounded up, i.e., 0.2 multiplied by 30 equals 6.

[0023] Specifically, the formula for calculating the drastic change in the deviation angle is: the drastic change in the deviation angle is equal to the sum of the absolute values ​​of the pitch angle differences between adjacent moments from the first pitch angle to the Nth pitch angle. A larger value for this drastic change in the deviation angle indicates more violent head movements and less stable attention among participants in the recent period.

[0024] Then, the shoulder orientation angle at the current moment is obtained, and the shoulder orientation angles of the M consecutive moments preceding the current moment are extracted from the shoulder orientation angle time series to calculate the shoulder orientation stability. M is a positive integer determined based on the body rotation response time and the video frame rate. The body rotation response time is determined based on the physiological delay and inertial characteristics of the human torso rotating from rest, and is usually taken as 0.5 seconds; the video frame rate is also taken as 30 frames per second. Optionally, M is equal to the body rotation response time multiplied by the video frame rate, and the result is rounded up, i.e., 0.5 multiplied by 30 equals 15.

[0025] Specifically, the calculation steps for shoulder orientation stability are as follows: First, calculate the mean and standard deviation of M consecutive shoulder orientation angles. Then, calculate the quotient of the mean divided by the standard deviation plus one. Adding one to the standard deviation is used to avoid a denominator of zero; when the standard deviation is zero, stability equals the mean. A higher calculated shoulder orientation stability indicates a more stable body orientation and a longer-lasting focus.

[0026] Furthermore, a gaze shift factor is calculated based on the instantaneous deviation angle and the drastic change in the deviation angle. The gaze shift factor equals the instantaneous deviation angle divided by 90 degrees, multiplied by the drastic change in the deviation angle, and divided by the reference drastic change. Here, 90 degrees is the maximum possible deviation angle of the head pitch angle relative to the digital human's line of sight, and the reference drastic change is the maximum possible sum of the absolute values ​​of the differences between adjacent pitch angles over N consecutive moments, i.e., N multiplied by the maximum single-step pitch angle change of 180 degrees. A larger gaze shift factor indicates a higher degree of gaze shift and lower attention level among participants.

[0027] Simultaneously, a body attention factor is calculated based on the shoulder orientation angle and shoulder orientation stability. Specifically, the body attention factor equals the shoulder orientation angle divided by the maximum orientation angle multiplied by the shoulder orientation stability. The shoulder orientation angle is defined as 0 degrees directly in front of the digital figure, with a maximum deviation angle of 90 degrees. A higher body attention factor indicates that the participant's body orientation is more directly facing the digital figure and their posture is more stable, indicating a stronger willingness to pay attention.

[0028] Finally, the social attention weight is obtained by calculating the weighted sum of the reciprocal of the gaze shift factor and the body attention factor. The reciprocal of the gaze shift factor converts the degree of gaze shift into positive attention; the larger the gaze shift factor, the smaller its reciprocal. The formula for calculating the weighted sum is: social attention weight equals the first weighting coefficient multiplied by the reciprocal of the gaze shift factor plus the second weighting coefficient multiplied by the body attention factor. The sum of the first and second weighting coefficients is 1. The specific values ​​can be set according to the importance of head posture and body orientation in the attention representation in the actual scenario. For example, the first weighting coefficient can be set to 0.6 and the second weighting coefficient to 0.4, making head posture information dominant. A higher social attention weight indicates a higher level of attention from the participants to the digital human, and it should receive a higher priority in subsequent gaze target ranking.

[0029] In summary, through the aforementioned multi-dimensional fusion calculations, social attention weights can comprehensively reflect the attention intentions and stability of participants in terms of head posture and body orientation. The digital human can perceive the attention intentions of participants from both head and body perspectives, avoiding misjudgments caused by relying solely on a single audio signal. Ultimately, through social attention weights, the digital human can accurately identify participants who are truly paying attention to it, avoiding focusing on distracting targets.

[0030] S30: Based on a continuous video frame image sequence, combined with the time sequence of key point coordinates on both sides of the shoulder, generate the corresponding shoulder key point displacement trajectory and direction change sequence, and calculate the interaction activity by combining the displacement amplitude and the number of direction changes. Furthermore, based on a continuous sequence of video frame images, combined with a temporal sequence of key point coordinates on both sides of the shoulders, a corresponding sequence of shoulder key point displacement trajectories and directional changes is generated. The temporal sequence of key point coordinates on both sides of the shoulders is a sequence of left and right shoulder key point coordinates inferred from each frame image, arranged in chronological order. Since the coordinates of the left and right shoulder key points change in real time with the participant's body movements such as swaying, leaning forward and backward, and turning left and right, the participant's limb activity level can be quantified by tracking the displacement trajectory and directional changes of the shoulder center point.

[0031] A direction change sequence is a time-series sequence recording the instantaneous displacement direction of each point on the displacement trajectory of the shoulder center point. It is used to characterize the change pattern of the participant's limb movement direction over time. Since participants' limb movements in video conferences typically exhibit various patterns such as swaying back and forth, swinging left and right, or pitching forward and backward, the displacement direction of the shoulder center point differs significantly between different patterns. By analyzing the difference between adjacent direction angles in the direction change sequence, it can be determined whether the participant is continuously engaging in meaningful interactive actions. For example, frequent changes in direction angles indicate active limb activity, while a constant direction angle over a long period indicates that the participant is in a relatively static state.

[0032] Specifically, a continuous sequence of video frame images is acquired, and combined with the temporal sequence of coordinates of key points on both sides of the shoulder, a corresponding sequence of displacement trajectories and directional changes of the shoulder key points is generated, including: Extract the coordinates of the left shoulder key point and the right shoulder key point of each frame from the time sequence of the key point coordinates on both sides of the shoulder, calculate the coordinates of the midpoint of the line connecting the left and right shoulder key points, and obtain the coordinate sequence of the shoulder center point. The time interval between two adjacent frames is extracted from a continuous video frame image sequence. Combined with the coordinate difference between two adjacent frames in the shoulder center point coordinate sequence, the displacement vector of the shoulder center point in each time interval is calculated to obtain the displacement vector sequence. The displacement vector sequence is connected in chronological order, with the starting position of the shoulder center point as the starting point and the displacement vectors of each frame connected end to end to generate the displacement trajectory of the shoulder center point. Using the timestamp corresponding to each point on the displacement trajectory of the shoulder center point as an index, the instantaneous displacement direction of the current point is recorded to obtain the direction change sequence.

[0033] First, the coordinates of the left and right shoulder keypoints in each frame are extracted from the temporal sequence of keypoint coordinates on both sides of the shoulder. The midpoint coordinates of the line connecting the left and right shoulder keypoints are then calculated to obtain the shoulder center point coordinate sequence. Since the left and right shoulder keypoints have stable detection results, and the midpoint of the line connecting the two shoulders can effectively represent the center position of the human torso, the coordinates of this midpoint change with the participant's body movements such as swaying, leaning forward, leaning back, and turning left and right. The midpoint coordinates calculated in each frame are arranged in chronological order to form the shoulder center point coordinate sequence, which reflects the trajectory of the shoulder center's position change over continuous time.

[0034] Secondly, the time interval between two adjacent frames is extracted from the continuous video frame image sequence. Combined with the coordinate difference between two adjacent frames in the shoulder center point coordinate sequence, the displacement vector of the shoulder center point within each time interval is calculated, resulting in a displacement vector sequence. Specifically, let the shoulder center point coordinates of the i-th frame be (x... i ,y i The coordinates of the shoulder center point in the (i+1)th frame are (x... i+1 ,y i+1 If the time interval between two adjacent frames is Δt, then the displacement vector from frame i to frame i+1 is (d x ,d y ) equals (x i+1 -x i ,y i+1 -y i This displacement vector reflects the distance and direction of movement of the shoulder center point within a time interval Δt. Repeating this calculation for all adjacent frames yields the displacement vector sequence.

[0035] Then, the displacement vector sequence is connected in chronological order, starting from the initial position of the shoulder center point, and the displacement vectors of each frame are connected end-to-end to generate the shoulder center point displacement trajectory. Specifically, starting from the first point of the shoulder center point coordinate sequence, the starting point of the first displacement vector is placed at this starting point, and the vector is moved along the direction of the displacement vector to obtain the end point of the first displacement vector; then the starting point of the second displacement vector is placed at the end point of the previous displacement vector, and the movement continues, and so on, until all displacement vectors are connected end-to-end to form a continuous broken line trajectory, which is the shoulder center point displacement trajectory. This shoulder center point displacement trajectory can intuitively reflect the movement path of the participant's shoulder center over continuous time.

[0036] Furthermore, using the timestamp corresponding to each point on the shoulder center point displacement trajectory as an index, the instantaneous displacement direction of the current point is recorded to obtain a sequence of direction changes. The instantaneous displacement direction is obtained by calculating the angle of the displacement vector corresponding to each point on the shoulder center point displacement trajectory. Specifically, for each timestamp corresponding to the displacement vector (d... x ,d y Its direction angle is calculated using the arctangent function, that is, the direction angle is equal to arctan2(d y ,d x The value ranges from negative π to π or from 0 degrees to 360 degrees. The directional angles corresponding to each timestamp are arranged in chronological order to form a sequence of directional changes.

[0037] This sequence of directional changes records the changes in the direction of the participant's shoulder center movement over time. The difference in the directional angle between adjacent time stamps reflects the change in movement direction; the more frequent the directional changes, the more active the participant's limb movements. Through the above steps, the participant's limb movements can be quantified as the displacement trajectory and directional change sequence of key shoulder points, providing a data foundation for subsequent calculations of interaction activity.

[0038] Furthermore, interaction activity is calculated by combining displacement amplitude and the number of directional changes. Displacement amplitude reflects the intensity of a participant's physical movements; a larger displacement amplitude indicates more vigorous movement. The number of directional changes reflects the frequency of a participant's physical movements; more frequent directional changes indicate more active movement. Interaction activity is a quantitative indicator that combines displacement amplitude and the number of directional changes, used to assess the level of participation and engagement of participants.

[0039] Specifically, the interaction activity is calculated by combining the displacement amplitude and the number of directional changes, including: The magnitude of each displacement vector in the displacement vector sequence is extracted to obtain the displacement amplitude sequence; The number of direction changes is obtained by counting the number of times that the directions of two adjacent instantaneous displacements are different from the sequence of direction changes. Extract the L consecutive displacement amplitudes before the current moment from the displacement amplitude sequence, and calculate the arithmetic mean of the L displacement amplitudes to obtain the average displacement amplitude, where L is a positive integer determined based on the limb action response time and the video frame rate; The interaction activity is calculated by dividing the average displacement amplitude by the unit length and combining it with the number of directional changes.

[0040] First, the magnitude of each displacement vector is extracted from the displacement vector sequence to obtain the displacement amplitude sequence. Specifically, for each displacement vector (d... x ,d y Its modulus is equal to d. x squared plus d yThe square root of the square reflects the distance the shoulder center point moves between two adjacent frames. Arranging the displacement amplitudes of each time step in chronological order forms a displacement amplitude sequence. A larger displacement amplitude indicates greater intensity of the participant's limb movements during that time period.

[0041] Secondly, the number of times the displacement directions of two adjacent instantaneous moments differ is counted from the direction change sequence to obtain the direction change count. The direction change sequence records the displacement direction angle corresponding to each timestamp. When the difference between two adjacent direction angles exceeds a preset direction change threshold, it is determined that the direction has changed. The preset direction change threshold represents the minimum angle difference required to determine a valid change in the movement direction of the shoulder center point. This preset direction change threshold is set based on the natural fluctuation range of the direction angle during slight shoulder swaying and the sensor measurement accuracy; for example, it can be set to 15 degrees. When the absolute value of the difference between two adjacent direction angles is greater than 15 degrees, it is determined that the direction has changed. A higher number of direction changes indicates more frequent body movements and higher interactive activity among participants.

[0042] Furthermore, the average displacement amplitude is obtained by extracting L consecutive displacement amplitudes prior to the current moment from the displacement amplitude sequence and calculating the arithmetic mean of these L displacement amplitudes. L is a positive integer determined based on the limb movement response time and the video frame rate. The limb movement response time is determined based on the minimum physiological delay required for the human body to produce noticeable limb movement in response to visual or auditory stimuli, typically taken as 0.3 seconds; the video frame rate is determined based on the real-time and smoothness requirements of the video conferencing scenario and the capabilities of common camera hardware, typically taken as 30 frames per second. Optionally, L is equal to the limb movement response time multiplied by the video frame rate, rounded up, i.e., 0.3 multiplied by 30 equals 9. The average displacement amplitude reflects the average intensity of the participants' limb movements within a recent time window. Compared to instantaneous displacement amplitude, the average displacement amplitude has better stability and anti-interference capability.

[0043] Furthermore, the interaction activity is calculated by dividing the average displacement amplitude by the unit length and combining this with the number of directional changes. The unit length is determined based on the pixel units of the shoulder keypoint coordinates in the video image, and is set to 1 pixel. This is used to perform dimensionless processing on the average displacement amplitude, eliminating the influence of pixel scale on the numerical value.

[0044] Specifically, the formula for calculating interaction activity is: Interaction activity equals the average displacement amplitude divided by the unit length, multiplied by the number of directional changes. The average displacement amplitude reflects the average intensity of the limb movement, and the number of directional changes reflects the frequency of the limb movement. The product of the two reflects the overall activity level of the participant's limb movements. The higher the interaction activity, the more active the participant's limb movements, the stronger their willingness to participate in the interaction, and the higher their priority should be in the subsequent ranking of visual targets.

[0045] In summary, through the above calculations, digital humans can quantify the interactive activity of participants from the perspective of body movements, providing a more comprehensive behavioral basis for target selection.

[0046] S40: Using the social attention weight and the interaction activity as multiplicative constraints, the instantaneous audio energy value of the corresponding participant is corrected to obtain the corrected energy value, and all participants are sorted from high to low according to the corrected energy value to generate a dynamic gaze target sequence. Furthermore, using social attention weight and interactive activity as multiplicative constraints, the instantaneous audio energy value of the corresponding participant is corrected to obtain the corrected energy value. The instantaneous audio energy value originates from the participant's audio input stream and reflects the participant's vocal intensity at the current moment. In multi-person meeting scenarios, relying solely on audio energy to drive eye contact can lead to abrupt eye contact switching during interruptions, simultaneous speaking by multiple participants, or silence periods. Therefore, social attention weight and interactive activity are introduced as correction factors. Social attention weight reflects the participant's level of attention to the digital avatar, while interactive activity reflects the participant's level of physical activity. The corrected energy value is obtained by multiplying the instantaneous audio energy value by the social attention weight and interactive activity. When a participant is both looking at the digital avatar and physically active while speaking, their corrected energy value is significantly amplified; when a participant speaks but is distracted or physically inactive, their corrected energy value is correspondingly suppressed.

[0047] Then, all participants are sorted from highest to lowest modified energy value, generating a dynamic gaze target sequence. A higher modified energy value indicates that the participant is both the current source of sound and paying attention to the digital human, while also exhibiting active body language, thus having the highest overall interaction priority. All participants are then arranged from highest to lowest modified energy value, forming the dynamic gaze target sequence. Participants ranked higher in this dynamic gaze target sequence should be the primary focus of the digital human's gaze. Because the modified energy value changes dynamically with time and participant behavior, the gaze target sequence is also dynamically updated, enabling the digital human's gaze to respond in real-time to changes in group behavior within the meeting setting.

[0048] Specifically, using the social attention weight and the interaction activity as multiplicative constraints, the instantaneous audio energy value of the corresponding participant is corrected to obtain a corrected energy value. All participants are then sorted from high to low according to the corrected energy value to generate a dynamic gaze target sequence, including: Based on the audio input stream of each participant, the instantaneous audio energy value of the corresponding participant at the current moment is obtained, and combined with the corresponding social attention weight and interaction activity, the instantaneous corrected energy value at the current moment is calculated. From the historical instantaneous corrected energy value sequence of each participant, extract the instantaneous corrected energy values ​​of the L consecutive moments before the current moment, and calculate the arithmetic mean of the L instantaneous corrected energy values ​​to obtain the smoothed corrected energy value; Iterate through all participants and collect the smoothed correction energy value of each participant to obtain a set of smoothed correction energy values. The values ​​in the smoothed correction energy value set are arranged in descending order, and the corresponding participant identifiers are extracted. A dynamic line-of-sight target sequence is generated according to the arrangement order.

[0049] First, based on the audio input stream of each participant, the instantaneous audio energy value of that participant at the current moment is obtained. Combined with the corresponding social attention weight and interaction activity level, the instantaneous corrected energy value at the current moment is calculated. The instantaneous audio energy value originates from the audio signal collected by the participant's microphone, reflecting the current vocal intensity. The social attention weight reflects the participant's level of attention to the digital human, and the interaction activity level reflects the participant's level of physical activity.

[0050] Specifically, the instantaneous corrected energy value is calculated as follows: the instantaneous audio energy value equals the instantaneous audio energy value multiplied by the social attention weight multiplied by the interaction activity level. When a participant simultaneously possesses high audio energy, high social attention weight, and high interaction activity level, their instantaneous corrected energy value is significantly amplified; when any dimension value is low, the instantaneous corrected energy value is correspondingly suppressed.

[0051] Secondly, from the historical instantaneous corrected energy value sequence of each participant, the instantaneous corrected energy values ​​of the L consecutive moments preceding the current moment are extracted, and the arithmetic mean of the L instantaneous corrected energy values ​​is calculated to obtain the smoothed corrected energy value. The historical instantaneous corrected energy value sequence is obtained by storing the instantaneous corrected energy values ​​calculated at each past moment in chronological order. L is a positive integer determined based on the limb movement response time and the video frame rate. The limb movement response time is usually taken as 0.3 seconds, and the video frame rate is usually taken as 30 frames per second. L equals 0.3 multiplied by 30 equals 9. The smoothed corrected energy value, by averaging the instantaneous corrected energy values ​​of multiple recent moments, can effectively suppress drastic jumps in the corrected energy value caused by instantaneous audio fluctuations or single-frame detection errors, making the update of the gaze target sequence smoother and more stable.

[0052] Then, the smoothed energy value of each participant is collected by iterating through all participants, resulting in a set of smoothed energy values. This set of smoothed energy values ​​contains the smoothed energy values ​​of all participants at the current moment, with each value corresponding to a participant identifier.

[0053] Subsequently, the values ​​in the smoothing correction energy value set are arranged in descending order, and the corresponding participant identifiers are extracted. A dynamic gaze target sequence is then generated according to the arrangement order. Participants with higher smoothing correction energy values ​​indicate higher interaction priority, which combines audio energy, social attention, and interactive activity, and should become the focus of the digital human's gaze earlier. The participant identifiers are arranged in descending order to form the dynamic gaze target sequence. Because the smoothing correction energy values ​​change dynamically over time, this sequence is also dynamically updated, enabling the digital human's gaze to respond in real time to changes in group behavior within the meeting scenario.

[0054] Through the aforementioned multiplicative constraint correction and dynamic ranking, the digital human's gaze control not only considers audio energy but also integrates two behavioral dimensions: social attention and interactive activity. This makes gaze target selection more consistent with the multi-factor decision-making in real interpersonal interactions. For example, when a participant is speaking but looking down at their phone, their social attention weight is low, the corrected energy value is suppressed, and the digital human will not look at them. Conversely, when a participant is not speaking but continuously looks at the digital human and engages in active body movements, their corrected energy value may exceed that of a speaker with low activity, enabling the digital human to pay attention to potentially high-engagement participants.

[0055] S50: Control the digital human's gaze focus to move between targets in the dynamic gaze target sequence through interpolation, obtain a smooth path between the current gaze focus and the next gaze target, and adjust the initial slope of the smooth path in combination with the social attention weight ratio of adjacent targets to generate a gaze movement trajectory.

[0056] Furthermore, the digital human's gaze focus is interpolated and shifted between targets in the dynamic gaze target sequence to obtain a smooth path between the current gaze focus and the next gaze target. The dynamic gaze target sequence indicates the order in which the digital human's gaze should focus on the participants, but directly jumping from one target to the next would result in an abrupt gaze shift, which does not conform to the physiological characteristics of smooth gaze movement in real interpersonal interaction. Therefore, interpolation is needed between the current gaze focus position and the next gaze target position to generate a smooth transition path. This interpolation can be achieved using Bézier curve fitting, setting intermediate control points between the start and end points to make the gaze focus's movement trajectory a smooth arc, rather than a straight jump.

[0057] Specifically, controlling the digital human's gaze focus to interpolate and shift between targets in the dynamic gaze target sequence to obtain a smooth path between the current gaze focus and the next gaze target includes: Use the current focal point of the line of sight as the starting point of the path, and the next target location of the line of sight as the ending point of the path; On the line connecting the starting point and the ending point of the path, the positions of two intermediate control points are determined, wherein the two intermediate control points are located on opposite sides of the line, and the offset distance is negatively correlated with the social attention weight ratio. A smooth path is generated by fitting a cubic Bézier curve, with the starting point of the path as the first control point, the ending point of the path as the fourth control point, and the two intermediate control points as the second and third control points.

[0058] First, the current gaze focus position is taken as the starting point of the path, and the next gaze target position is taken as the ending point. The current gaze focus position is the coordinate of the participant that the digital human is currently looking at in the image coordinate system, and the next gaze target position is the coordinate of the shoulder center point or face center point of the next participant to be focused on in the dynamic gaze target sequence. The gaze transfer from the starting point to the ending point needs to generate a smooth movement trajectory.

[0059] The step of determining the location of the next line-of-sight target includes: Read the first participant in the dynamic line-of-sight target sequence after the participant corresponding to the current line-of-sight focus, and use it as a candidate target participant; Extract the current shoulder center point coordinates of the candidate target participants from the shoulder center point coordinate sequence, and use them as the reference coordinates; Extract the shoulder center point coordinates of the candidate target participant for L consecutive moments before the current moment from the shoulder center point coordinate sequence, and calculate the arithmetic mean to obtain the smoothed center coordinates; Calculate the Euclidean distance between the reference coordinates and the smoothed center coordinates. When the Euclidean distance is less than a distance threshold, the smoothed center coordinates are used as the next line of sight target position; otherwise, the reference coordinates are used as the next line of sight target position. The distance threshold is determined based on the standard deviation of the fluctuation of the shoulder center point coordinates in a seated state.

[0060] Specifically, the steps for determining the location of the next line of sight target include the following:

[0061] First, the participant immediately following the participant at the current line of sight is selected from the dynamic line-of-sight target sequence as a potential target participant. The dynamic line-of-sight target sequence sorts all participants from highest to lowest modified energy value, and the participant immediately following the participant at the current line of sight is the next potential target to focus on.

[0062] Secondly, the current shoulder center point coordinates of the candidate participant are extracted from the shoulder center point coordinate sequence and used as the reference coordinates. The shoulder center point coordinates are calculated by the midpoint of the line connecting the left and right shoulder key points, reflecting the real-time position of the participant's shoulder center in the image. The current shoulder center point coordinates are the most direct measurement of the current position of the candidate participant.

[0063] Furthermore, the shoulder center point coordinates of the candidate participant for the previous L consecutive moments are extracted from the shoulder center point coordinate sequence, and their arithmetic mean is calculated to obtain the smoothed center coordinates. L is a positive integer determined based on the limb movement response time and the video frame rate. The limb movement response time is typically 0.3 seconds, and the video frame rate is typically 30 frames per second; therefore, L equals 0.3 multiplied by 30, which equals 9. The smoothed center coordinates, by averaging the shoulder center point coordinates over L consecutive moments, effectively suppress coordinate fluctuations caused by minor swaying of the participant or posture estimation noise, reflecting the stable position of the participant's shoulder center.

[0064] Furthermore, the Euclidean distance between the reference coordinates and the smoothed center coordinates is calculated. When the Euclidean distance is less than a distance threshold, the smoothed center coordinates are used as the next line-of-sight target position; otherwise, the reference coordinates are used. Specifically, the reference coordinates reflect the real-time position at the current moment, while the smoothed center coordinates reflect the recent average position. When the Euclidean distance between the reference coordinates and the smoothed center coordinates is small, it indicates that the participant's shoulder center position is relatively stable. In this case, using the smoothed center coordinates as the line-of-sight target position can further improve the stability of the target position. When the Euclidean distance is large, it indicates that the participant has undergone significant body displacement or posture changes. In this case, the reference coordinates should be used as the line-of-sight target position to ensure the line-of-sight target's responsiveness to rapid changes.

[0065] The distance threshold is determined based on the standard deviation of the fluctuation of the shoulder center point coordinates while seated. Specifically, the standard deviation of the Euclidean distance between the shoulder center point coordinates of multiple participants while seated is statistically analyzed, and 2 to 3 times the standard deviation is taken as the distance threshold. For example, if the standard deviation of the shoulder center point coordinate fluctuation while seated is 5 pixels, the distance threshold can be set to 10 to 15 pixels. This threshold setting method ensures that smooth coordinates are used to maintain stability within the normal range of seated swaying, while real-time coordinates are used to ensure response speed when actual displacement occurs. Through the above determination steps, the target position of the line of sight can maintain stability and respond promptly to the actual body movements of the participants.

[0066] Furthermore, the positions of two intermediate control points are determined on the line connecting the starting and ending points of the path. These two intermediate control points are located on opposite sides of the connecting line, and their offset distance is negatively correlated with the social attention weight ratio. Specifically, let the starting point of the path be A, the ending point be B, the length of the connecting line be L, and the social attention weight ratio be R. The formula for calculating the offset distance H is: H = (L / 4) × (1 / (R+1)). The two intermediate control points are located on opposite sides of the perpendicular bisector of the connecting line, both at a distance H from the line. Specifically, in the direction of the connecting line, the first intermediate control point is located at one-third of the distance from A to B, and the second intermediate control point is located at two-thirds of the distance from A to B, then offset by positive and negative H respectively along the perpendicular direction of the connecting line. This formula ensures that the larger R is, the smaller H is, making the path closer to a straight line and the gaze shift faster; the smaller R is, the larger H is, the greater the curvature of the path and the smoother the gaze shift.

[0067] Furthermore, a cubic Bézier curve fitting method is used, with the starting point of the path as the first control point, the ending point of the path as the fourth control point, and the two intermediate control points as the second and third control points, to generate a smooth path.

[0068] A Bézier curve is a smooth parametric curve defined by control points, the shape of which is determined by the position of those control points. Cubic Bézier curves are used for fitting because they achieve the best balance between computational complexity and curve freedom: a linear Bézier curve is a straight line and cannot achieve curved transitions; a quadratic Bézier curve has only one intermediate control point, resulting in a single curvature and making it difficult to independently control the tangent direction at both the start and end points; while a cubic Bézier curve has two intermediate control points, allowing for independent adjustment of the start and end directions. Furthermore, the computational cost of higher-order curves is far greater than that of cubic curves. Therefore, cubic Bézier curves are sufficient to simulate the smooth, curved trajectory required for natural line-of-sight movement while also meeting real-time requirements.

[0069] Specifically, a cubic Bézier curve is defined by four control points: the curve starts from the first control point, extends in the direction from the first control point to the second control point, turns to the direction of the third control point, and finally reaches the fourth control point.

[0070] Specifically, the mathematical expression for a cubic Bézier curve is: in, Parameters between 0 and 1 , , , These are the first, second, third, and fourth control points, respectively. The curve starts from... Depart, towards Extend in direction, then turn Finally arrived The curve always lies within the convex hull formed by the four control points, and is consistent with... and Tangent. Specifically, the fitting method: given the starting point of the path. (First control point) and endpoint (Fourth control point), and two intermediate control points. , By directly substituting the values ​​into the above equation and taking a series of t values ​​(e.g., t=0, 0.1, 0.2, ..., 1), the coordinates of the points on the curve can be calculated. Connecting these points sequentially yields the smooth path. In actual control, smooth movement can be achieved simply by calculating the discrete points on the curve and driving the line of sight focus to pass through these points sequentially.

[0071] By fitting a cubic Bézier curve, a smooth curve trajectory from the starting point to the ending point can be generated, allowing the digital human's gaze focus to move along this curve, achieving a natural gaze shift. Simultaneously, through the aforementioned interpolation method, the curvature and speed of the gaze trajectory can be adaptively adjusted according to the social attention weight ratio of adjacent targets, making the gaze shift behavior more consistent with the gaze shift patterns in real interpersonal interactions.

[0072] Furthermore, the initial slope of the smooth path is adjusted by combining the social attention weight ratio of adjacent targets to generate the gaze movement trajectory. Social attention weight reflects the level of attention participants pay to the digital human. When the digital human's gaze shifts from one participant to another, the two participants have different social attention weights, and the speed and trajectory of the gaze movement should differ accordingly. If the social attention weight of the next target is significantly higher than that of the current target, it indicates that the digital human needs to respond more quickly to the interaction needs of the high-attention target. In this case, the initial slope of the gaze movement trajectory should be greater, causing the gaze to accelerate away from the current target and quickly approach the next target. If the social attention weight of the next target is lower, the gaze movement can be relatively smoother.

[0073] Specifically, the initial slope of the smooth path is adjusted by combining the social attention weight ratio of adjacent targets to generate a gaze movement trajectory, including: Obtain the first, second, third, and fourth control points of the cubic Bézier curve corresponding to the smooth path; Calculate the ratio of the social attention weight of the participant corresponding to the next gaze target to the social attention weight of the participant corresponding to the current gaze focus, and use it as the weight ratio; Calculate the vector pointing from the first control point to the second control point, multiply the magnitude of the vector by the weight ratio, and keep the direction unchanged to obtain the correction vector; Add the first control point to the correction vector to obtain a new second control point; A cubic Bézier curve is regenerated using the first control point, the new second control point, the third control point, and the fourth control point, as the line-of-sight movement trajectory after adjusting the initial slope; The digital human's gaze focus is controlled to move from the first control point to the fourth control point according to the adjusted gaze movement trajectory.

[0074] First, obtain the first, second, third, and fourth control points of the cubic Bézier curve corresponding to the smooth path. The first control point is the current focal point, i.e., the starting point of the path; the fourth control point is the next target location, i.e., the ending point of the path; and the second and third control points are two previously determined intermediate control points. These four control points collectively define the shape of the cubic Bézier curve.

[0075] Secondly, the ratio of the social attention weight of the participant corresponding to the next visual target to the social attention weight of the participant corresponding to the current visual focus is calculated as the weight ratio. This weight ratio reflects the degree of social attention paid to the next target relative to the current target. When the weight ratio is greater than 1, it indicates that the next target is more attention-grabbing than the current target, and the gaze shift should be faster and more direct; when the weight ratio is less than 1, it indicates that the next target has lower attention, and the gaze shift can be relatively gradual.

[0076] Next, the vector pointing from the first control point to the second control point is calculated. The magnitude of this vector is multiplied by the weight ratio, while keeping the direction unchanged, to obtain the corrected vector. This corrected vector is obtained by subtracting the x-coordinate of the first control point from the x-coordinate of the second control point, and subtracting the y-coordinate of the first control point from the y-coordinate of the second control point. Keeping the direction unchanged means only changing the length of the vector without altering its pointing angle; that is, multiplying the x-coordinate and y-coordinate of the original vector by the weight ratio respectively, resulting in a new vector with the same direction as the original vector.

[0077] Specifically, when the weight ratio is greater than 1, the magnitude of the correction vector increases, the second control point moves away from the first control point along the original direction, making the tangent direction of the initial segment of the Bézier curve longer, the initial slope increases, and the line of sight moves away from the current target faster; when the weight ratio is less than 1, the magnitude of the correction vector decreases, the initial slope decreases, and the line of sight shifts more smoothly.

[0078] Furthermore, the first control point is added to the correction vector to obtain a new second control point. The coordinates of the new second control point are equal to the x-coordinate of the first control point plus the x-component of the correction vector, and the y-coordinate of the first control point plus the y-component of the correction vector. By adjusting the position of the second control point, the slope of the initial segment of the Bézier curve is corrected.

[0079] A cubic Bézier curve was regenerated using the first control point, a new second control point, a third control point, and a fourth control point, serving as the line-of-sight trajectory after adjusting the initial slope. The new curve kept the start and end points unchanged, only adjusting the positions of the control points in the initial segment, thereby altering the direction and speed characteristics of the curve's initial segment.

[0080] Finally, the digital human's gaze focus moves from the first control point to the fourth control point according to the adjusted gaze trajectory. The digital human's gaze focus moves smoothly along the regenerated Bézier curve, achieving a gaze shift from the current target to the next target. Through the above-mentioned adjustment of the initial slope, when the next target has a high social attention weight, the gaze accelerates away from the current target and quickly moves to the high-attention target; when the next target has a low social attention weight, the gaze transitions smoothly, avoiding visual discomfort caused by frequent and rapid shifts. This step allows the initial slope of the gaze trajectory to adaptively adjust according to the difference in social attention weight between adjacent targets, improving the naturalness of gaze control and the ability to convey social signals.

[0081] In summary, through the aforementioned interpolation movement and slope adjustment, the digital human's gaze movement trajectory not only achieves a smooth transition of gaze focus between different participants, but also adaptively adjusts the movement speed according to the differences in social attention weights of adjacent targets. This makes the gaze switching behavior more consistent with the gaze shift patterns in real interpersonal interactions, enhancing the immersiveness and realism of the interaction between the digital human and the user.

[0082] In summary, the embodiments of this application have at least the following technical effects: This invention first identifies multiple participants in a video conference scenario and extracts behavioral parameter sets by responding to multiple video stream inputs, obtaining rich nonverbal social cues from a visual dimension, providing multi-dimensional decision-making basis for gaze control. Second, it calculates the gaze direction deviation angle based on the head pitch angle time sequence and combines it with the shoulder orientation angle to obtain social attention weights, quantifying the representation of attention intention by head posture and body orientation into calculable weight factors, enabling the digital human to perceive who is paying attention to it. Third, it calculates interaction activity based on the shoulder key point displacement trajectory and direction change sequence, incorporating the activity level of body movements into the scope of gaze target selection, enabling the digital human to identify which participants have a stronger willingness to interact.

[0083] Finally, the audio energy value is corrected by multiplicatively constraining social attention weight and interaction activity to generate a dynamic gaze target sequence. Natural and coherent gaze movement trajectories are generated by interpolation movement and smooth path adjustment. This solves the problems of stiff gaze switching and ignoring non-verbal social cues in traditional methods, and improves the naturalness of digital human interaction and user reach in multi-person meeting scenarios.

[0084] Example 2, as Figure 2 As shown, based on the same inventive concept as the digital human gaze control method combining group behavior recognition provided in Embodiment 1, this embodiment of the invention also provides a digital human gaze control system combining group behavior recognition, including: The identification and data acquisition module 11 is used to respond to multiple video stream inputs, identify multiple participants in the video conference scene where the digital human is located as candidate interaction objects, and extract the behavioral parameter set of each participant; The attention weight calculation module 12 is used to calculate the deviation angle between the gaze direction of the corresponding participant and the gaze direction of the digital human based on the head pitch angle time sequence of each participant, and to obtain the social attention weight based on the deviation angle and the shoulder orientation angle. The interaction activity calculation module 13 is used to generate the corresponding shoulder key point displacement trajectory and direction change sequence based on the continuous video frame image sequence and the time sequence of the coordinates of key points on both sides of the shoulder, and to calculate the interaction activity by combining the displacement amplitude and the number of direction changes. The gaze target sequence generation module 14 is used to correct the instantaneous audio energy value of the corresponding participant by using the social attention weight and the interaction activity as multiplicative constraints, to obtain the corrected energy value, and to sort all participants from high to low according to the corrected energy value to generate a dynamic gaze target sequence. The gaze movement trajectory generation module 15 is used to control the gaze focus of the digital human to move between each target in the dynamic gaze target sequence through interpolation, obtain a smooth path between the current gaze focus and the next gaze target, and adjust the initial slope of the smooth path in combination with the social attention weight ratio of adjacent targets to generate a gaze movement trajectory.

[0085] Specifically, the identification and data acquisition module 11 is used for: In response to multiple video stream inputs, the system identifies multiple participants in the video conferencing scenario where the digital human is located as candidate interaction objects and extracts the behavioral parameter set of each participant.

[0086] Specifically, the behavioral parameter set includes at least the head pitch angle time sequence, the shoulder orientation angle time sequence, and the coordinate time sequence of key points on both sides of the shoulder.

[0087] The attention weight calculation module 12 is specifically used for: Based on the time sequence of each participant's head pitch angle, the deviation angle between the corresponding participant's gaze direction and the digital human's line of sight is calculated. Social attention weights are then obtained based on the deviation angle and the shoulder orientation angle, including: Extract the current head pitch angle from the head pitch angle time sequence, calculate the absolute value of the angle between the head pitch angle and the direction of the digital human's line of sight, and obtain the instantaneous deviation angle; The head pitch angles of the N consecutive moments preceding the current moment are extracted from the head pitch angle time sequence. The deviation angle change severity is calculated by combining the pitch angle difference between adjacent moments, where N is a positive integer determined based on the head motion response time and the video frame rate. Obtain the shoulder orientation angle at the current moment, and extract the shoulder orientation angles of the M consecutive moments before the current moment from the shoulder orientation angle time sequence to calculate the shoulder orientation stability, where M is a positive integer determined based on the body rotation response time and the video frame rate; Based on the instantaneous deviation angle and the drastic change of the deviation angle, a gaze offset factor is calculated; Based on the shoulder orientation angle and the shoulder orientation stability, calculate the body attention factor; The social attention weight is obtained by calculating the weighted sum of the reciprocal of the gaze shift factor and the body attention factor.

[0088] The interaction activity calculation module 13 is specifically used for: Obtain a continuous sequence of video frame images, and combine it with the temporal sequence of coordinates of key points on both sides of the shoulder to generate the corresponding displacement trajectory and direction change sequence of the shoulder key points, including: Extract the coordinates of the left shoulder key point and the right shoulder key point of each frame from the time sequence of the key point coordinates on both sides of the shoulder, calculate the coordinates of the midpoint of the line connecting the left and right shoulder key points, and obtain the coordinate sequence of the shoulder center point. The time interval between two adjacent frames is extracted from a continuous video frame image sequence. Combined with the coordinate difference between two adjacent frames in the shoulder center point coordinate sequence, the displacement vector of the shoulder center point in each time interval is calculated to obtain the displacement vector sequence. The displacement vector sequence is connected in chronological order, with the starting position of the shoulder center point as the starting point and the displacement vectors of each frame connected end to end to generate the displacement trajectory of the shoulder center point. Using the timestamp corresponding to each point on the displacement trajectory of the shoulder center point as an index, the instantaneous displacement direction of the current point is recorded to obtain the direction change sequence.

[0089] Specifically, the interaction activity is calculated by combining the displacement amplitude and the number of directional changes, including: The magnitude of each displacement vector in the displacement vector sequence is extracted to obtain the displacement amplitude sequence; The number of direction changes is obtained by counting the number of times that the directions of two adjacent instantaneous displacements are different from the sequence of direction changes. Extract the L consecutive displacement amplitudes before the current moment from the displacement amplitude sequence, and calculate the arithmetic mean of the L displacement amplitudes to obtain the average displacement amplitude, where L is a positive integer determined based on the limb action response time and the video frame rate; The interaction activity is calculated by dividing the average displacement amplitude by the unit length and combining it with the number of directional changes.

[0090] The line-of-sight target sequence generation module 14 is specifically used for: Using the social attention weight and the interaction activity as multiplicative constraints, the instantaneous audio energy value of the corresponding participant is corrected to obtain a corrected energy value. All participants are then sorted from high to low according to the corrected energy value to generate a dynamic gaze target sequence, including: Based on the audio input stream of each participant, the instantaneous audio energy value of the corresponding participant at the current moment is obtained, and combined with the corresponding social attention weight and interaction activity, the instantaneous corrected energy value at the current moment is calculated. From the historical instantaneous corrected energy value sequence of each participant, extract the instantaneous corrected energy values ​​of the L consecutive moments before the current moment, and calculate the arithmetic mean of the L instantaneous corrected energy values ​​to obtain the smoothed corrected energy value; Iterate through all participants and collect the smoothed correction energy value of each participant to obtain a set of smoothed correction energy values. The values ​​in the smoothed correction energy value set are arranged in descending order, and the corresponding participant identifiers are extracted. A dynamic line-of-sight target sequence is generated according to the arrangement order.

[0091] The line-of-sight movement trajectory generation module 15 is specifically used for: Controlling the digital human's gaze focus to interpolate and shift between targets in the dynamic gaze target sequence to obtain a smooth path between the current gaze focus and the next gaze target includes: Use the current focal point of the line of sight as the starting point of the path, and the next target location of the line of sight as the ending point of the path; On the line connecting the starting point and the ending point of the path, the positions of two intermediate control points are determined, wherein the two intermediate control points are located on opposite sides of the line, and the offset distance is negatively correlated with the social attention weight ratio. A smooth path is generated by fitting a cubic Bézier curve, with the starting point of the path as the first control point, the ending point of the path as the fourth control point, and the two intermediate control points as the second and third control points.

[0092] The step of determining the location of the next line-of-sight target includes: Read the first participant in the dynamic line-of-sight target sequence after the participant corresponding to the current line-of-sight focus, and use it as a candidate target participant; Extract the current shoulder center point coordinates of the candidate target participants from the shoulder center point coordinate sequence, and use them as the reference coordinates; Extract the shoulder center point coordinates of the candidate target participant for L consecutive moments before the current moment from the shoulder center point coordinate sequence, and calculate the arithmetic mean to obtain the smoothed center coordinates; Calculate the Euclidean distance between the reference coordinates and the smoothed center coordinates. When the Euclidean distance is less than a distance threshold, the smoothed center coordinates are used as the next line of sight target position; otherwise, the reference coordinates are used as the next line of sight target position. The distance threshold is determined based on the standard deviation of the fluctuation of the shoulder center point coordinates in a seated state.

[0093] Furthermore, the initial slope of the smooth path is adjusted by combining the social attention weight ratio of adjacent targets to generate a gaze movement trajectory, including: Obtain the first, second, third, and fourth control points of the cubic Bézier curve corresponding to the smooth path; Calculate the ratio of the social attention weight of the participant corresponding to the next gaze target to the social attention weight of the participant corresponding to the current gaze focus, and use it as the weight ratio; Calculate the vector pointing from the first control point to the second control point, multiply the magnitude of the vector by the weight ratio, and keep the direction unchanged to obtain the correction vector; Add the first control point to the correction vector to obtain a new second control point; A cubic Bézier curve is regenerated using the first control point, the new second control point, the third control point, and the fourth control point, as the line-of-sight movement trajectory after adjusting the initial slope; The digital human's gaze focus is controlled to move from the first control point to the fourth control point according to the adjusted gaze movement trajectory.

Claims

1. A digital human gaze control method combining group behavior recognition, characterized in that: The method includes: Responding to multiple video stream inputs, the system identifies multiple participants in the video conferencing scenario where the digital human is located as candidate interaction objects and extracts the behavioral parameter set of each participant. Based on the time sequence of each participant's head pitch angle, the deviation angle between the corresponding participant's gaze direction and the digital human's line of sight is calculated, and the social attention weight is obtained based on the deviation angle and the shoulder orientation angle. Based on a continuous sequence of video frame images, combined with the temporal sequence of coordinates of key points on both sides of the shoulder, the corresponding displacement trajectory and direction change sequence of the shoulder key points are generated, and the interaction activity is calculated by combining the displacement amplitude and the number of direction changes. Using the social attention weight and the interaction activity as multiplicative constraints, the instantaneous audio energy value of the corresponding participant is corrected to obtain the corrected energy value. All participants are then sorted from high to low according to the corrected energy value to generate a dynamic gaze target sequence. The digital human's gaze focus is controlled to interpolate and move between targets in the dynamic gaze target sequence to obtain a smooth path between the current gaze focus and the next gaze target. The initial slope of the smooth path is adjusted by combining the social attention weight ratio of adjacent targets to generate a gaze movement trajectory.

2. The digital human gaze control method combining group behavior recognition according to claim 1, characterized in that, The behavioral parameter set includes at least the head pitch angle time sequence, the shoulder orientation angle time sequence, and the coordinate time sequence of key points on both sides of the shoulder.

3. The digital human gaze control method combining group behavior recognition according to claim 1, characterized in that, Based on the time sequence of each participant's head pitch angle, the deviation angle between the corresponding participant's gaze direction and the digital human's line of sight is calculated. Social attention weights are then obtained based on the deviation angle and the shoulder orientation angle, including: Extract the current head pitch angle from the head pitch angle time sequence, calculate the absolute value of the angle between the head pitch angle and the direction of the digital human's line of sight, and obtain the instantaneous deviation angle; The head pitch angles of the N consecutive moments preceding the current moment are extracted from the head pitch angle time sequence. The deviation angle change severity is calculated by combining the pitch angle difference between adjacent moments, where N is a positive integer determined based on the head motion response time and the video frame rate. Obtain the shoulder orientation angle at the current moment, and extract the shoulder orientation angles of the M consecutive moments before the current moment from the shoulder orientation angle time sequence to calculate the shoulder orientation stability, where M is a positive integer determined based on the body rotation response time and the video frame rate; Based on the instantaneous deviation angle and the drastic change of the deviation angle, a gaze offset factor is calculated; Based on the shoulder orientation angle and the shoulder orientation stability, calculate the body attention factor; The social attention weight is obtained by calculating the weighted sum of the reciprocal of the gaze shift factor and the body attention factor.

4. The digital human gaze control method combining group behavior recognition according to claim 1, characterized in that, Obtain a continuous sequence of video frame images, and combine it with the temporal sequence of coordinates of key points on both sides of the shoulder to generate the corresponding displacement trajectory and direction change sequence of the shoulder key points, including: Extract the coordinates of the left shoulder key point and the right shoulder key point of each frame from the time sequence of the key point coordinates on both sides of the shoulder, calculate the coordinates of the midpoint of the line connecting the left and right shoulder key points, and obtain the coordinate sequence of the shoulder center point. The time interval between two adjacent frames is extracted from a continuous video frame image sequence. Combined with the coordinate difference between two adjacent frames in the shoulder center point coordinate sequence, the displacement vector of the shoulder center point in each time interval is calculated to obtain the displacement vector sequence. The displacement vector sequence is connected in chronological order, with the starting position of the shoulder center point as the starting point and the displacement vectors of each frame connected end to end to generate the displacement trajectory of the shoulder center point. Using the timestamp corresponding to each point on the displacement trajectory of the shoulder center point as an index, the instantaneous displacement direction of the current point is recorded to obtain the direction change sequence.

5. The digital human gaze control method combining group behavior recognition according to claim 1, characterized in that, The interaction activity is calculated by combining the displacement amplitude and the number of directional changes, including: The magnitude of each displacement vector in the displacement vector sequence is extracted to obtain the displacement amplitude sequence; The number of direction changes is obtained by counting the number of times that the directions of two adjacent instantaneous displacements are different from the sequence of direction changes. Extract the L consecutive displacement amplitudes before the current moment from the displacement amplitude sequence, and calculate the arithmetic mean of the L displacement amplitudes to obtain the average displacement amplitude, where L is a positive integer determined based on the limb action response time and the video frame rate; The interaction activity is calculated by dividing the average displacement amplitude by the unit length and combining it with the number of directional changes.

6. The digital human gaze control method combining group behavior recognition according to claim 1, characterized in that, Using the social attention weight and the interaction activity as multiplicative constraints, the instantaneous audio energy value of the corresponding participant is corrected to obtain a corrected energy value. All participants are then sorted from high to low according to the corrected energy value to generate a dynamic gaze target sequence, including: Based on the audio input stream of each participant, the instantaneous audio energy value of the corresponding participant at the current moment is obtained, and combined with the corresponding social attention weight and interaction activity, the instantaneous corrected energy value at the current moment is calculated. From the historical instantaneous corrected energy value sequence of each participant, extract the instantaneous corrected energy values ​​of the L consecutive moments before the current moment, and calculate the arithmetic mean of the L instantaneous corrected energy values ​​to obtain the smoothed corrected energy value; Iterate through all participants and collect the smoothed correction energy value of each participant to obtain a set of smoothed correction energy values. The values ​​in the smoothed correction energy value set are arranged in descending order, and the corresponding participant identifiers are extracted. A dynamic line-of-sight target sequence is generated according to the arrangement order.

7. The digital human gaze control method combining group behavior recognition according to claim 1, characterized in that, Controlling the digital human's gaze focus to interpolate and shift between targets in the dynamic gaze target sequence to obtain a smooth path between the current gaze focus and the next gaze target includes: Use the current focal point of the line of sight as the starting point of the path, and the next target location of the line of sight as the ending point of the path; On the line connecting the starting point and the ending point of the path, the positions of two intermediate control points are determined, wherein the two intermediate control points are located on opposite sides of the line, and the offset distance is negatively correlated with the social attention weight ratio. A smooth path is generated by fitting a cubic Bézier curve, with the starting point of the path as the first control point, the ending point of the path as the fourth control point, and the two intermediate control points as the second and third control points.

8. The digital human gaze control method combining group behavior recognition according to claim 7, characterized in that, The steps for determining the location of the next line-of-sight target include: Read the first participant in the dynamic line-of-sight target sequence after the participant corresponding to the current line-of-sight focus, and use it as a candidate target participant; Extract the current shoulder center point coordinates of the candidate target participants from the shoulder center point coordinate sequence, and use them as the reference coordinates; Extract the shoulder center point coordinates of the candidate target participant for L consecutive moments before the current moment from the shoulder center point coordinate sequence, and calculate the arithmetic mean to obtain the smoothed center coordinates; Calculate the Euclidean distance between the reference coordinates and the smoothed center coordinates. When the Euclidean distance is less than a distance threshold, the smoothed center coordinates are used as the next line of sight target position; otherwise, the reference coordinates are used as the next line of sight target position. The distance threshold is determined based on the standard deviation of the fluctuation of the shoulder center point coordinates in a seated state.

9. The digital human gaze control method combining group behavior recognition according to claim 1, characterized in that, The initial slope of the smooth path is adjusted by combining the social attention weight ratio of adjacent targets to generate a gaze movement trajectory, including: Obtain the first, second, third, and fourth control points of the cubic Bézier curve corresponding to the smooth path; Calculate the ratio of the social attention weight of the participant corresponding to the next gaze target to the social attention weight of the participant corresponding to the current gaze focus, and use it as the weight ratio; Calculate the vector pointing from the first control point to the second control point, multiply the magnitude of the vector by the weight ratio, and keep the direction unchanged to obtain the correction vector; Add the first control point to the correction vector to obtain a new second control point; A cubic Bézier curve is regenerated using the first control point, the new second control point, the third control point, and the fourth control point, as the line-of-sight movement trajectory after adjusting the initial slope; The digital human's gaze focus is controlled to move from the first control point to the fourth control point according to the adjusted gaze movement trajectory.

10. A digital human gaze control system incorporating group behavior recognition, characterized in that: The digital human gaze control method combining group behavior recognition as described in any one of claims 1-9 includes: The identification and data acquisition module is used to respond to multiple video stream inputs, identify multiple participants in the video conference scene where the digital human is located as candidate interaction objects, and extract the behavioral parameter set of each participant; The attention weight calculation module is used to calculate the deviation angle between the gaze direction of each participant and the gaze direction of the digital human based on the head pitch angle time sequence of each participant, and to obtain the social attention weight based on the deviation angle and the shoulder orientation angle. The interaction activity calculation module is used to generate the corresponding shoulder key point displacement trajectory and direction change sequence based on the continuous video frame image sequence and the time sequence of the coordinates of key points on both sides of the shoulder, and to calculate the interaction activity by combining the displacement amplitude and the number of direction changes. The gaze target sequence generation module is used to correct the instantaneous audio energy value of the corresponding participant by using the social attention weight and the interaction activity as multiplicative constraints, to obtain the corrected energy value, and to sort all participants from high to low according to the corrected energy value to generate a dynamic gaze target sequence. The gaze movement trajectory generation module is used to control the digital human's gaze focus to move between targets in the dynamic gaze target sequence through interpolation, obtain a smooth path between the current gaze focus and the next gaze target, and adjust the initial slope of the smooth path by combining the social attention weight ratio of adjacent targets to generate the gaze movement trajectory.

Images