Synchronous control method of robot social behavior driven by scene information and neural modulation mechanism

Abstract

This paper discloses a method for synchronously controlling robot social behavior based on scene information and a neural modulation mechanism. The method first determines the social distance of the robots through the group's social space, then determines the robot's body rotation angle through optimal social interaction cohesion. Finally, based on sound source information, the optimal runtime is calculated under given boundary conditions to control eye-head coordination. This method has been validated on the Xiaopang robot platform and has demonstrated good results in coordinating social behavior in multi-object scenarios. An optimal control algorithm based on minimal neural transmission noise is used to modulate eye-head gaze behavior. Analysis of the dynamic characteristics of the robots under this model and strategy demonstrates their effectiveness and stability.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a method for synchronously controlling robot social behavior based on scene information and a neural modulation mechanism. Background Art

[0002] Robots are now widely used in a variety of scenarios, where they are expected to adhere to social protocols and exhibit natural behaviors. However, despite the prevalence of social interactions in daily life, comprehensive analyses of human social responses are rare. Enabling robots to understand social scenarios and exhibit natural human behaviors has become a widespread concern in modern society. Two key challenges remain: selection between different objects and the synchronized control of coordinated limbs. To address these challenges, two approaches have been explored: data-driven and model-driven. However, without prior knowledge of biological mechanisms, data-driven models struggle to explain human behavior on dynamic and long-term scales. Regarding model-driven approaches, many studies have focused on the fundamental coordination patterns of human behavior. Against this backdrop, this paper focuses on developing a model to represent coordinated social behavior, including body movement / orientation, head rotation, and eye movements. A synchronized control approach based on social space and neural modulation is proposed. This approach has been validated on the Xiaochubby robot platform and demonstrates promising results in representing coordinated social behavior in multi-object scenarios. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this paper presents a method for synchronized control of robot social behavior based on scene information and neural modulation mechanisms. This method addresses the control of robots in multi-target social scenarios and models coordinated behaviors, including body movement and orientation, head rotation, and eye movements. Based on this model, a synchronized control method driven by social space theory and neural modulation mechanisms is proposed. This method controls the robot's body according to the dynamic social space and regulates eye-head coordinate gaze behavior based on the law of minimum neural transmission noise.

[0004] This paper aims to model coordinated social behaviors in multi-target scenarios and proposes a synchronized control strategy for robot social behaviors, including body movement / orientation, head rotation, and eye movements. This strategy collects RGB-D images and sound fields perceived in the scene, determines body movement and orientation based on the social space and relative positions of the robot and human, and employs an optimal control algorithm based on minimal neural transmission noise to determine eye-head gaze behavior.

[0005] To achieve the above objectives, the present invention relates to a method for synchronously controlling robot social behavior based on scene information and a neural modulation mechanism, which specifically includes the following steps:

[0006] (1) Confirming the social distance of robots through the social space of the group

[0007] Social space is divided into two categories, namely personal space and group space. For a social individual, his personal space is an area centered on himself, which is represented by the Gaussian function f i p (x,y) to construct,

[0008] f i p The calculation process of (x,y) is as shown in formula (2a)-(2d)

[0009]

[0010] θ i =atan2((yy i ),(xx i ))(2b)

[0011]

[0012] For a group, the Gaussian function of the personal space of social objects is accumulated to obtain the Gaussian equation of the group space f i g (x,y)

[0013]

[0014] Among them, (x, y) is the position of the robot in the plane coordinate system, (x i ,y i ) is the position of social object i in the group in the plane coordinate system, A is the amplitude, σ x is the standard deviation in the horizontal direction, σ y is the standard deviation in the front-back direction, θ i is the deflection angle between the robot and the social object’s body, n is the number of social objects in the group;

[0015] Change the robot's position, thereby changing the robot's f i g (x,y) values, which enable the robot to maintain an appropriate distance from the group, i.e., a distance that satisfies the Hall social criterion;

[0016] (2) Confirming the robot's body turning angle through optimal social interaction cohesion

[0017] In multi-object scenarios, the social interaction cohesion score is used to represent the interaction intention in the group. The social interaction cohesion value is obtained by the following equation,

[0018]

[0019] S g =W g *n (6)

[0020]

[0021] S total =S i +S g +S p (8)

[0022] W p +W g +W i =1 (9)

[0023] Among them, θ ij is the angle between the body direction vectors of social objects i and j in the group on the XY plane of the plane coordinate system, W i represents the proportion of social interaction cohesion score to total cohesion, and G represents the social space f of the group. i g (x,y),W g Indicates the group size cohesion score S g Total cohesion score S total The proportion of W p is close to the cohesion score S p Total cohesion score S total The weight of dist(i,j) is the Euclidean distance between social objects i and j in the group, W p is the weight of the close cohesion score in the total cohesion. (x j ,y j ) is the position of social object j in the group in the plane coordinate system, (x i ,y i ) The position of social object i in the group in the plane coordinate system;

[0024] Find the body rotation angle when the social interaction cohesion score is the largest, and then obtain the angular relationship between the robot's body rotation angle and the body orientation of each social object under the optimal social group cohesion condition;

[0025] (3) Based on the sound source information, the optimal running time is solved under given boundary conditions to control the eye-head coordination.

[0026] Compared with existing technologies, this invention offers the following advantages: It establishes a model representing coordinated social behavior from the perspective of social cues and biology, and proposes a synchronous control method based on social space and neural modulation. This method has been validated on the Xiaochubby robot platform and demonstrates promising results in coordinating social behavior in multi-object scenarios. It also employs an optimal control algorithm based on minimal neural transmission noise to modulate eye-gaze behavior. Analysis of the robot's dynamic characteristics under this model and strategy demonstrates its effectiveness and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of the method for synchronous control of robot social behavior driven by scene information and neural modulation mechanism involved in the present invention.

[0028] Figure 2 It is the Gaussian function distribution diagram of personal space.

[0029] Figure 3 is the Gaussian function distribution diagram of the population space.

[0030] Figure 4 Schematic diagram of the Euclidean coordinate space for gaze behavior of eye-head coordination.

[0031] Figure 5 This is a schematic diagram of the working process of the robot testing platform involved in Example 2.

[0032] Figure 6 This is a diagram of the actual application scenario involved in Example 2. DETAILED DESCRIPTION

[0033] In order to more clearly illustrate the content of the present invention, the present invention is further described below in conjunction with the accompanying drawings and specific embodiments:

[0034] Example 1

[0035] In multi-target scenarios, a robot's behavior manifests itself in three key areas: maintaining social distance from social objects; adjusting body direction; and controlling eye-head coordination to maintain gaze on specific targets. These behaviors are driven by human stimuli and performed synchronously in real-world social situations. These three aspects coordinate to form a method for synchronized control of robot social behavior based on scene information and neural modulation mechanisms. Specifically, the following steps are involved:

[0036] 1. Confirm the robot’s social distance through the group’s social space

[0037] Social distance can be interpreted as the distance between two individuals, between an individual and a group, or between an internal and external group. Given that spatial perception plays a dominant role in regulating social behavior, we propose that social space is divided into two categories: personal space and group space. For a social individual (robot or human), their personal space is the area centered on themselves.

[0038] like Figure 2 As shown, the gradient inside the individual space is set to f i p (x,y). During the interaction, f i p (x,y) varies with the relative positions of social objects, which mimics the sense of space in social scenes, where A,σ x ,σ y are the standard deviations of amplitude, horizontal and front-back directions respectively, then the size of the individual dynamic social space depends on A, σ x ,σ y . It can be noted that the robot responds differently to the front and back interactions of the body, so we can replace σ y Divided into σ rear and σ front , where σ rear and σ front is the standard deviation in the front-back direction. The red line is f i p The minimum threshold that (x,y) can produce is usually regarded as the boundary of acceptable social distance. The comfortable space is usually constructed by using a Gaussian function, which sets the position of the person as the center of the Gaussian function and assigns a value to the space around the person to describe the degree to which the area accepts others to enter. Therefore, it adopts a two-dimensional asymmetric Gaussian function to implement f i p (x,y).

[0039] f i p (x,y)=AsymGauss(x,y,x i ,y i ,θ i ,A,σ x ,σ y ) (1)

[0040] σ x The value of is the standard deviation in the horizontal direction, and σ y The value of is the standard deviation in the front-back direction, (x, y) is the position of the robot in the plane coordinate system, (x i ,y i ) The position of social objects in a plane coordinate system. Their values ​​will change frequently based on people’s positions, postures, and social cues. Figure 3 As shown in Figure 2, when several people stand in front of the robot, the group space is the combination of their personal spaces, and the boundary of the group space surrounds all individual spaces. and is the personal space occupied by each of the three people. The green arrows indicate the direction the person and the robot are facing, and the angle (θ i ) is the deflection angle formed by the body directions of the two social objects (robot and group social object).

[0041] f i p The calculation process of (x,y) is as shown in formulas (2a)-(2d).

[0042]

[0043] θ i =atan2((yy i ),(xx i ))(2b)

[0044]

[0045] At the same time, based on the group space, we also need to meet the needs of personal space. It can be obtained by summing the Gaussian function of the personal space of social objects to obtain the Gaussian equation of the group space. Where n is the number of social objects in the group.

[0046]

[0047] In this way, we can obtain a group social space that satisfies the combination of personal space, and then we can change the position of the robot, and then change the robot's f i g The (x,y) value enables the robot to maintain an appropriate distance with the social object group, that is, a distance that satisfies the Hall social criterion.

[0048] 2. Confirm the robot's body turning angle through optimal cohesion

[0049] To obtain appropriate body orientation, in-group cohesion is used, which measures people using a multi-scale cohesion criterion.

[0050] Close cohesion score (S p ):S p It is derived from the proximity principle, as defined in equations (4) and (5).

[0051]

[0052] Where dist(i,j) is the Euclidean distance between social objects i and j in the group, W p is the weight of the close cohesion score to the total cohesion score, (x j ,y j ) is the position of social object j in the group in the plane coordinate system, (x i ,y i ) is the position of social object i in the group in the plane coordinate system.

[0053] Group size cohesion score (S g ): For a given social group, cohesion should be related to the number of groups. In this paper, we consider group cohesion to be proportional to group size, as defined by equation (6).

[0054] S g =W g *n (6)

[0055] Among them, W g Indicates the proportion of group size cohesion score to the total cohesion score

[0056] Social interaction cohesion score (S i ): In multi-object scenarios, the spatial relationship between any two individuals must be considered. Since the interaction intention in a group can be applied to all pairs, we use the social interaction cohesion score to represent it. The cohesion value is obtained by Equation (7).

[0057]

[0058] Among them, θ ij is the angle between the body direction vectors of social objects i and j in the group on the XY plane of the camera coordinates, W i represents the proportion of the social interaction cohesion score to the total cohesion score. It assumes that when two people are face to face, the cohesion value has a positive effect, otherwise it is negative. Therefore, we choose 1+cosθ ij Serves as the main operator to moderate the social interaction cohesion score.

[0059] Based on the above multi-scale representation, define S total For S p 、S g and S i By adjusting the normalized weights of the three scores (W p ,W g ,W i ), we can achieve a comprehensive description of dynamic social scenes.

[0060] S total =Si +S g +S p (8)

[0061] W p +W g +W i =1 (9)

[0062] Among them, S total Represents the total cohesion score.

[0063] In order to obtain the optimal total cohesion score (S total ), we need to obtain the maximum total score of social interaction cohesion (S i ), since the social interaction cohesion obtained by the robot at different body angles is different, we can use the algorithm to traverse and find the body angle when the social interaction cohesion score is the largest, so that we can obtain the angular relationship between the robot's body angle and the body orientation of each social object under the condition of optimal social group cohesion.

[0064] 3. Solve the optimal running time to control eye-head coordination under given boundary conditions

[0065] For robots in social scenarios, gaze behavior is the main manifestation of their establishing interactive relationships with specific people. When this interaction is extended to multi-object conditions, it is inevitable that a large amount of gaze shifts from one object to another is required. For example, when a robot is interacting with a social object, another person enters the robot's field of view with strong social intentions. The robot needs to respond appropriately to the latecomer, including shifting attention and gaze. The natural gaze model proposed in this paper is as follows: Figure 4 As shown, this gaze shifting behavior requires coordination between the eyes, head, and even body. It also needs to take into account vertical rotation across the Y axis and horizontal rotation along the Z axis, which is caused by the head twist (θ H and θ V ). Due to the involvement of the head, the gaze shift angle in a given direction is divided into the head rotation angle θ H ,θ H and focal line deflection angle α H , α V , and α L and α R is the horizontal angle of the eyeball in social gaze behavior. Figure 4 In the coordinate system XYZ and X H Y H Z H Represents the absolute coordinate system and the robot head coordinate system respectively. X′ H and Y′ H The axes represent X H Axis and YH Projection of the axis onto the horizontal plane XOY.

[0066] Multi-object interactions often involve social attention shifting behaviors where gaze shifts from one object to another. Gaze shifting is a process performed by a 2-DOF system consisting of eye movements and head rotations. The deviation of gaze shift is defined as the standard error of the noise, as shown in Equation (10).

[0067]

[0068] Among them, σ eye ,σ head represents the deviation of eye and head movement, t p is the duration of the gaze shift, H eye (t) is the impulse response function of eye movement, H head (t) Impulse response function of head movement. Assume E(t p ) and nerve control signals to the head and eyes (s head (t) and s eye (t)) are proportional to each other, and their coefficients are A and B respectively. eye Applied torque and focal line y eye The motion of is defined as formula (11a), where T eye is the muscle torque of eye movement. The muscles and orbital tissue have two lag times, whose constants are τ1 and τ2 respectively. We assume that T eye is the neural control signal eye A low-pass filter with a time constant τ e describes the delay between the muscle response and the control signal, as defined in Equation (11b).

[0069]

[0070] State vector represents the acceleration, velocity, and position of the focal line. The dynamics of the eyeball can be expressed in a third-order form, as shown in equations (12) and (13):

[0071]

[0072] As for the head, it is a rigid body controlled by vestibular and coelomic reflexes and is defined as a second-order system in Equation (14a).

[0073]

[0074] State vector represents the acceleration, velocity and torsion of the head. The muscle torque T of the head movement head is the control signal s head (The time constant is τh ) as shown in equations (15) and (16).

[0075]

[0076] Among them, K, V, and S represent the inertia, viscosity, and stiffness of the head movement. Through the above equations, we can obtain the impulse response function H eye (t) and H head (t), t represents the time of execution.

[0077]

[0078] in a3=(2K-Vτ h ) / 2Kτ h In order to solve the optimized control signal with the minimum variance of neural noise, the present invention defines the standard error of neural noise in equation (19)

[0079]

[0080] By solving the corresponding Hamiltonian function h eye (y e ,s eye ,t), we can get the optimal control signal [t0,t f ], taking the unidirectional eye dynamics model as an example, the loss function can be converted to Optimal signal s eye Can be calculated And set it to zero for semi-analytical solution, as in Equation (21), the initial and final conditions are given by and Given, get y eye The velocity distribution and t f The relationship between the numerical method and the f is the optimal control time, and the solution formula is shown in (20-22).

[0081]

[0082] is a vector of Lagrange multipliers. The above optimal solution method can be applied to the horizontal or vertical movement of the focal line deflection respectively. For the head, the above solution method is the same, as shown in formulas (23-25).

[0083]

[0084] where λ head is the vector of Lagrange multipliers. Through the above formula, we can get yhead The velocity distribution and t f The above method can be used to solve the optimal running time of the eye and head models under a given rotation angle when they are running independently. By linearly fitting the above two corresponding relationships, the running time t f , t f It will be used as a known condition to support the solution of eye-head coordinated control.

[0085] Example 2

[0086] 1. Integration with robot platform

[0087] The Xiaopang robot was used as the test platform. The Xiaopang robot is a social robot designed for educational and entertainment applications. It is equipped with multiple joint motors that enable movement, rotation, and head rotation. A screen is mounted on the robot's face to support Android-based animated projection, displaying a three-dimensional uniform surface with horizontal and vertical eye movements. For sensors, an RGB-D camera (Realsense D435) is mounted on the robot's head and connected to an internal NUC computer. During interaction, the robot's NUC computer receives data streams from the camera and microphone array, such as the 3D coordinates of the social target's face and the location of the sound source in the robot's coordinate system. This data is used to guide the robot's selection of social targets and behavioral expression. Spatial perception is then implemented using a facial recognition algorithm to determine the robot's positional relationship with the social target and the target's selection. Based on these selections, the robot animates its body, neck, and face to produce gaze behavior directed at the selected person. Simultaneously, the system's positional information is recorded, enabling its spatial position to be derived for practical application scenarios.

[0088] 2. Actual application scenarios

[0089] In actual social scenarios, robot social interactions often occur in interactions between multiple people and a robot. In this case, the robot first adjusts the distance between itself and the multiple people, as well as its own body rotation angle, to achieve a suitable state, as shown in Figure (6a). On the basis of maintaining social distance and body orientation, gaze behavior is taken into account. As shown in Figure (6b), initially, the robot maintains its gaze on People-1 by controlling the head movement and animated face on the screen. Then, People-2 will make some sounds that can be measured by the robot, causing the robot's social attention to shift. With this shift, the robot will shift its gaze behavior from People-1 to People-2. We can see that the robot's gaze behavior towards the social object is completed by eye saccades and head rotations, thus achieving the purpose of the robot's gaze shift in the scenario of multiple social objects.

Claims

1. A method for synchronously controlling robot social behavior based on scene information and neural modulation mechanism, characterized by: The specific steps include: (1) Confirming the social distance of robots through the social space of the group Social space is divided into two categories, namely personal space and group space. For a social individual, his personal space is an area centered on himself, which is represented by the Gaussian function. To build, The calculation process is as shown in formula (2a)-(2d) For a group, the Gaussian function of the personal space of social objects is accumulated to obtain the Gaussian equation of the group space. Among them, (x, y) is the position of the robot in the plane coordinate system, (x i ,y i ) is the position of social object i in the group in the plane coordinate system, A is the amplitude, σ x is the standard deviation in the horizontal direction, σ y is the standard deviation in the front-back direction, θ i is the deflection angle between the robot and the social object’s body, n is the number of social objects in the group; Change the robot's position, thereby changing the robot's value, so that the robot and the group can maintain a suitable distance, that is, a distance that satisfies the Hall social criterion; (2) Confirming the robot's body turning angle through optimal social interaction cohesion In multi-object scenarios, the social interaction cohesion score is used to represent the interaction intention in the group. The social interaction cohesion value is obtained by the following equation, S total =S i +S g +S p #(8) IN p +W g +W i =1#(9) Among them, θ ij is the angle between the body direction vectors of social objects i and j in the group on the XY plane of the plane coordinate system, W i represents the proportion of social interaction cohesion score to total cohesion, and G represents the social space of the group W g Indicates the group size cohesion score S g Total cohesion score S total The proportion of W p is close to the cohesion score S p Total cohesion score S total The weight of dist(i,j) is the Euclidean distance between social objects i and social objects j in the group, (x j ,y j ) is the position of social object j in the group in the plane coordinate system, (x i ,y i ) The position of social object i in the group in the plane coordinate system, S i score social interaction cohesion; Find the body rotation angle when the social interaction cohesion score is the largest, and then obtain the angular relationship between the robot's body rotation angle and the body orientation of each social object under the optimal social group cohesion condition; (3) Based on the sound source information, the optimal running time is solved under given boundary conditions to control the eye-head coordination.

Images