A method and device for human-robot interaction for a character robot

Abstract

The present application relates to a kind of human-computer interaction method and device for character robot, comprising: obtaining the multi-source information in target area;According to multi-source information, determine the behavior mode of pedestrian in target area;Based on behavior mode, determine the emotional feedback mode of character robot;Determine the feedback action corresponding to emotional feedback mode, and obtain the action parameter corresponding to feedback action, control instruction is generated according to action parameter, control instruction is used to control character robot to execute feedback action.The present application is fused to judge the current interactive subject behavior to multi-source information;According to interactive subject behavior, call the feedback action corresponding to it, so that the interactive response of character robot is more appropriate, and then improve the interactive effect of character robot.

Description

Technical Field

[0001] This invention relates to the field of natural human-computer interaction, and more particularly to a human-computer interaction method and apparatus for role-playing robots. Background Technology

[0002] In emerging fields such as digital entertainment, stage performances, spatial experiences, educational displays, and new media art, typical interactive application scenarios for role-playing robots have been proposed. The key characteristic of these scenarios is that role-playing robots, through interactions with the audience (humans), the environment (objects), and intelligent agents (machines), bring multimodal sensory narratives and impacts, thereby conveying information and emotions within the human-machine-object triadic world. It is evident that role-playing robots capable of providing positive interactive feedback to the audience are crucial in these interactive application scenarios.

[0003] However, the currently constructed character robots are mainly for display performances. These character robots execute actions designed by designers through pre-written motion programs, resulting in serious homogenization and a lack of innovation, and the interactive feedback provided to the audience does not meet expectations.

[0004] Therefore, it is urgent to redesign the interactive actions of the character robot in order to improve its interactive capabilities and enhance the appeal of the interactive scenes to the audience. Summary of the Invention

[0005] The purpose of this invention is to provide a human-computer interaction method for character robots, in order to solve the problem that character robots in the prior art cannot provide good interactive feedback to the audience, so as to enable character robots to make interactive responses that are more in line with the audience's behavior.

[0006] In a first aspect, the present invention provides a human-computer interaction method for a role-playing robot, the method comprising:

[0007] Acquire multi-source information within the target area;

[0008] Based on the multi-source information, determine the pedestrian behavior patterns within the target area;

[0009] Based on the behavioral pattern, the emotional feedback pattern of the robot is determined;

[0010] The feedback action corresponding to the emotional feedback mode is determined, and the action parameters corresponding to the feedback action are obtained. A control command is generated according to the action parameters, and the control command is used to control the character robot to execute the feedback action.

[0011] According to the human-computer interaction method for role-playing robots provided by the present invention, acquiring multi-source information within a target area specifically includes:

[0012] Multi-source information within the target area is collected using multi-source sensors.

[0013] The human-computer interaction method for a role-playing robot provided by the present invention utilizes multi-source sensors to collect multi-source information within the target area, specifically including at least two of the following:

[0014] The multi-source sensor includes an image sensor, which is used to acquire scene images of the target area; the target area is a region within a certain range surrounding the robot.

[0015] The multi-source sensor includes a distance sensor, which is used to collect distance information between the robot and pedestrians within the target area.

[0016] The multi-source sensor includes a pressure sensor, which is used to collect pressure information of the robot.

[0017] The multi-source sensor includes a vibration sensor, which utilizes the vibration information of the robot.

[0018] According to the human-computer interaction method for a role robot provided by the present invention, the scene image of the target area, the distance information between the role robot and the pedestrian in the target area, the pressure information of the role robot, and the vibration information of the role robot are used as the multi-source information.

[0019] Determining the pedestrian behavior pattern within the target area based on the multi-source information includes:

[0020] Perform facial recognition on the scene image of the target area;

[0021] If the robot's pressure / vibration information indicates that it is under pressure / vibration, then the touch / tapping Boolean value will be recorded as positive; otherwise, the touch / tapping Boolean value will not be recorded as positive.

[0022] Based on the facial recognition results, the distance information between the robot and the pedestrian in the target area, the touch Boolean value, and the tap Boolean value, the behavior pattern of the pedestrian in the target area is determined.

[0023] According to the human-computer interaction method for a role-playing robot provided by the present invention, determining the behavior pattern of a pedestrian in the target area based on facial recognition results, distance information between the role-playing robot and a pedestrian within the target area, the touch Boolean value, and the tap Boolean value includes:

[0024] If the face recognition result is that no human figure image envelope is detected, then the behavior mode is the unmanned interaction mode;

[0025] If the human image recognition result is that at least one human image envelope is detected and all detected human image envelopes meet the preset conditions, then the behavior mode is the pass-through mode.

[0026] If the human image recognition result is that multiple human image envelopes are identified and at least two of the identified human image envelopes do not meet the preset conditions, then the behavior mode is a multi-person interaction mode.

[0027] If the human image recognition result is that a human figure image envelope is detected, the detected human figure image envelope does not meet the preset conditions, and the touch Boolean value is positive, then the behavior mode is touch mode;

[0028] If the human image recognition result is that a human figure image envelope is detected, the detected human figure image envelope does not meet the preset conditions, and the tapping Boolean value is positive, then the behavior mode is the tapping mode.

[0029] If the human image recognition result is that a human figure image envelope is detected, the detected human figure image envelope does not meet the preset conditions, the spatial position of the detected human figure image envelope only changes within the preset spatial range, the touch Boolean value is not positive and the tap Boolean value is not positive, then the behavior mode is the gaze mode.

[0030] The variation in the spatial position of the envelope of the identified human figure image is determined by the distance information between the robot and the pedestrian within the target area;

[0031] The preset condition is that the time it takes for the human-shaped image envelope to pass through the robot is less than a preset time.

[0032] According to the human-computer interaction method for role-playing robots provided by the present invention, the priority of the behavior pattern is recorded in a behavior pattern priority sequence;

[0033] Determining the emotional feedback pattern of the robot based on the behavioral pattern includes:

[0034] If the behavior mode is not the multi-person interaction mode, the emotional feedback mode associated with the behavior mode is searched from the pre-stored behavior mode-emotional feedback mode lookup table, and the found emotional feedback mode is used as the emotional feedback mode of the character robot.

[0035] When the behavior pattern is the multi-person interaction pattern, the behavior pattern of each pedestrian participating in the interaction is refined, and the behavior pattern with the highest priority is selected.

[0036] The system searches for the emotional feedback pattern associated with the highest priority behavior pattern in the pre-stored behavior pattern-emotional feedback pattern lookup table, and uses the found emotional feedback pattern as the emotional feedback pattern of the robot.

[0037] According to the human-computer interaction method for a role-playing robot provided by the present invention, determining the feedback action corresponding to the emotional feedback pattern includes:

[0038] Find the corresponding feedback action for the emotional feedback pattern from the pre-stored emotional feedback pattern-feedback action lookup table;

[0039] The step of obtaining the action parameters corresponding to the feedback action includes:

[0040] The motion parameters corresponding to the feedback motion are retrieved from the pre-stored motion library; the motion library stores each feedback motion designed for the robot character and its corresponding motion parameters.

[0041] The emotional feedback modes include: boredom mode, fright mode, curiosity mode, clinginess mode, and excitement mode;

[0042] The feedback action is a combination of interactive actions of different body parts of the character robot.

[0043] According to the human-computer interaction method for a role-playing robot provided by the present invention, the process of generating the action parameters corresponding to each feedback action includes:

[0044] Determine the interactive actions included in each of the feedback actions;

[0045] The motion parameters corresponding to each interactive action in each type of feedback action are generated using 3D modeling technology and animation retargeting technology.

[0046] Interpolation processing is performed on the action parameters corresponding to each interactive action to enable the robot to execute the corresponding interactive action smoothly and fluently.

[0047] Based on the action parameters corresponding to each interactive action after interpolation, the interactive actions in each feedback action are arranged to obtain the action parameters corresponding to each feedback action.

[0048] In the motion simulation system, verify the motion parameters corresponding to each feedback action. If the verification result shows that the control command generated according to the motion parameters corresponding to each feedback action can control the robot to execute each feedback action smoothly, output the motion parameters corresponding to each feedback action. If the verification result shows that the control command generated according to the motion parameters corresponding to each feedback action cannot control the robot to execute each feedback action smoothly, repeat the above operation.

[0049] According to the human-computer interaction method for a role-playing robot provided by the present invention, the step of generating motion parameters corresponding to each interactive action in each feedback action using 3D modeling technology and animation retargeting technology includes:

[0050] Based on 3D modeling technology, a robot animation model with a virtual animated skeleton is generated according to the number of joints, position, and degrees of freedom of the robot character.

[0051] The motion animation corresponding to each interactive action is generated using motion capture or animation software editing methods.

[0052] Based on the motion animation corresponding to each interactive action and the robot animation model, animation retargeting technology is used to generate motion parameters corresponding to each interactive action that can be executed on the character robot.

[0053] Secondly, the present invention also provides a human-computer interaction device for a role-playing robot, the device comprising:

[0054] The acquisition module is used to acquire multi-source information within the target area;

[0055] The behavior pattern determination module is used to determine the behavior pattern of pedestrians within the target area based on the multi-source information.

[0056] An emotion feedback pattern determination module is used to determine the emotion feedback pattern of the role robot based on the behavior pattern.

[0057] An interaction module is used to determine the feedback action corresponding to the emotional feedback mode, obtain the action parameters corresponding to the feedback action, and generate control instructions according to the action parameters. The control instructions are used to control the character robot to execute the feedback action.

[0058] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the human-computer interaction method for a role-playing robot as described in the first aspect.

[0059] In a fourth aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the human-computer interaction method for a role-playing robot as described in the first aspect.

[0060] This invention provides a human-computer interaction method and apparatus for a role-playing robot. The method involves acquiring multi-source information within a target area; determining the behavior patterns of pedestrians within the target area based on the multi-source information; determining the emotional feedback pattern of the role-playing robot based on the behavior patterns; determining the feedback action corresponding to the emotional feedback pattern and acquiring the action parameters corresponding to the feedback action; generating control instructions according to the action parameters; and using the control instructions to control the role-playing robot to execute the feedback action. This invention fuses multi-source information to determine the current interaction subject's behavior; and invokes the corresponding feedback action based on the interaction subject's behavior to make the role-playing robot's interaction response more appropriate, thereby improving the interaction effect of the role-playing robot. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0062] Figure 1 This is a flowchart illustrating the human-computer interaction method for role-playing robots provided by the present invention;

[0063] Figure 2 This is a schematic diagram of the method for generating action parameters corresponding to interactive actions provided by the present invention;

[0064] Figure 3 This is a structural schematic diagram of the human-computer interaction device for role-playing robots provided by the present invention;

[0065] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0067] The following is combined with Figures 1 to 4This invention describes a human-computer interaction method for role-playing robots.

[0068] For role-playing robots used in public demonstrations, the requirements for motion accuracy, load characteristics, and sensing are roughly the same as those for industrial robots and service robots. In addition, role-playing robots need to consider motion characteristics, interactive feedback, and action implementation as closely as possible to the set role attributes and interactive context, achieving natural human-machine interaction and preserving the motion characteristics of the role attributes. To make the motion characteristics of the role-playing robot closer to the design goals, the structural design of the role-playing robot usually needs to consider the implementation of multi-degree-of-freedom and complex-degree-of-freedom movements. This undoubtedly increases the difficulty of human-machine interaction. Therefore, this invention provides a human-machine interaction method for role-playing robots, such as... Figure 1 As shown, the method includes:

[0069] S11. Obtain multi-source information within the target area;

[0070] It is understandable that multi-source information refers to the general term for different types of information obtained from different information sources.

[0071] S12. Determine the pedestrian behavior patterns within the target area based on the multi-source information;

[0072] When processing multi-source information, a robot's interactive feedback cannot simply be a one-to-one response, as this would result in a stiff and mechanical interaction. Therefore, it is necessary to fuse multi-source information to determine the current interactive subject's behavior. For example, when a robot senses a touch or tap from the outside world, it can simultaneously determine the location and age of the interactive subject based on image information, thereby making a more appropriate interactive response.

[0073] S13. Based on the behavioral pattern, determine the emotional feedback pattern of the robot.

[0074] The types of behavioral patterns and the associated emotional feedback patterns have been determined during the design phase and can be queried during actual application.

[0075] S14. Determine the feedback action corresponding to the emotional feedback mode, obtain the action parameters corresponding to the feedback action, generate control instructions according to the action parameters, and use the control instructions to control the character robot to perform the feedback action.

[0076] Understandably, motion parameters are parameters that control the rotation angle and speed of the robot's joints.

[0077] In terms of interactive feedback, the robot's own dynamics are primarily used to provide anthropomorphic action feedback. For example, combinations of head and tail movements, as well as the overall body posture, are used to express the robot's current emotional characteristics.

[0078] This invention provides a human-computer interaction method for a role-playing robot, which fuses multi-source information to determine the current interaction subject's behavior; and calls corresponding feedback actions based on the interaction subject's behavior to make the role-playing robot's interaction response more appropriate, thereby improving the role-playing robot's interaction effect.

[0079] Based on the above embodiments, as an optional embodiment, acquiring multi-source information within the target area specifically includes:

[0080] Multi-source information within the target area is collected using multi-source sensors.

[0081] For role-playing robots, various sensors serve as windows for interaction with the outside world. Role-playing robots utilize sensors deployed on their bodies to acquire multi-source information (such as image information, voice information, touch information, and tapping information), then convert physical information such as sound, light, vibration, and pressure into electrical signals, which are input into the robot's control system. This allows the control system to provide different feedback based on the programmed settings.

[0082] Based on the above embodiments, as an optional embodiment, multi-source information within the target area is collected using a multi-source sensor, specifically including at least two of the following:

[0083] The multi-source sensor includes an image sensor, which is used to acquire scene images of the target area; the target area is a region within a certain range surrounding the robot.

[0084] The multi-source sensor includes a distance sensor, which is used to collect distance information between the robot and pedestrians within the target area.

[0085] The multi-source sensor includes a pressure sensor, which is used to collect pressure information of the robot.

[0086] The multi-source sensor includes a vibration sensor, which utilizes the vibration information of the robot.

[0087] In addition to the aforementioned sensors, the multi-source sensors also include image sensors, voice sensors (such as microphone arrays), and light sensors.

[0088] The following describes several types of interactive devices that can be installed on a character robot;

[0089] Image sensors (cameras): Acquire various visual information, which can be used for face and gesture detection. Using depth cameras or dual-camera systems can obtain imprecise positional information of objects in front, better recognizing body movements and enabling richer interactions.

[0090] Microphone array: Detects ambient sound information, and combined with speech recognition and dialogue systems, it enables human-computer voice interaction.

[0091] Pressure sensor: It can detect pressure signals on the robot's "skin" to recognize tactile information such as touching and pressing by the interactive subject, and realize limb interaction.

[0092] Vibration sensors: These detect vibrations around the robot, supplementing pressure sensors and enhancing tactile interaction. They can also be installed around the robot to detect human approach or tapping, among other interactive behaviors.

[0093] Distance sensor: detects whether an object is approaching the robot, and is generally used to determine close-range interaction behavior.

[0094] Light sensor: It senses changes in the brightness of the robot's surrounding environment, enabling some special interactive behaviors.

[0095] It should be noted that the required multi-source sensors vary depending on the different interaction scenarios of the robot, and adjustments should be made appropriately according to the actual situation.

[0096] Based on the above embodiments, as an optional embodiment, the scene image of the target area, the distance information between the robot and the pedestrian in the target area, the pressure information of the robot, and the vibration information of the robot are used as the multi-source information;

[0097] Determining the pedestrian behavior pattern within the target area based on the multi-source information includes:

[0098] Perform facial recognition on the scene image of the target area;

[0099] If the robot's pressure / vibration information indicates that it is under pressure / vibration, then the touch / tapping Boolean value will be recorded as positive; otherwise, the touch / tapping Boolean value will not be recorded as positive.

[0100] Based on the facial recognition results, the distance information between the robot and the pedestrian in the target area, the touch Boolean value, and the tap Boolean value, the behavior pattern of the pedestrian in the target area is determined.

[0101] In this embodiment, the scene image of the target area, the distance information between the robot and the pedestrian within the target area, the pressure information of the robot, and the vibration information of the robot are used as multi-source information for determining the pedestrian's behavior pattern. Based on this, the scene image of the target area, the pressure information of the robot, and the vibration information of the robot are preprocessed to obtain preprocessed data.

[0102] This invention fuses multi-source information to determine the current interactive subject's behavior, laying the foundation for generating interactive responses of role-playing robots.

[0103] Based on the above embodiments, as an optional embodiment, determining the pedestrian's behavior pattern within the target area based on facial recognition results, distance information between the robot and the pedestrian within the target area, the touch Boolean value, and the tap Boolean value includes:

[0104] If the face recognition result is that no human figure image envelope is detected, then the behavior mode is the unmanned interaction mode;

[0105] If the human image recognition result is that at least one human image envelope is detected and all detected human image envelopes meet the preset conditions, then the behavior mode is the pass-through mode.

[0106] If the human image recognition result is that multiple human image envelopes are identified and at least two of the identified human image envelopes do not meet the preset conditions, then the behavior mode is a multi-person interaction mode.

[0107] If the human image recognition result is that a human figure image envelope is detected, the detected human figure image envelope does not meet the preset conditions, and the touch Boolean value is positive, then the behavior mode is touch mode;

[0108] If the human image recognition result is that a human figure image envelope is detected, the detected human figure image envelope does not meet the preset conditions, and the tapping Boolean value is positive, then the behavior mode is the tapping mode.

[0109] If the human image recognition result is that a human figure image envelope is detected, the detected human figure image envelope does not meet the preset conditions, the spatial position of the detected human figure image envelope only changes within the preset spatial range, the touch Boolean value is not positive and the tap Boolean value is not positive, then the behavior mode is the gaze mode.

[0110] The variation in the spatial position of the envelope of the identified human figure image is determined by the distance information between the robot and the pedestrian within the target area;

[0111] The preset condition is that the time it takes for the human-shaped image envelope to pass through the robot is less than a preset time.

[0112] Because the main input for the interaction of the role-playing robot is the behavior of pedestrians / spectators in public places, the purpose of the interaction is to attract the attention of pedestrians / spectators through anthropomorphic role-playing interaction design, increase the participation of the interactive display and spontaneous multi-user interaction, and increase the dwell time in public places. Therefore, this invention classifies the behavior states of pedestrians in public scenes. Based on pedestrian behavior observation, the states of pedestrians can be categorized as no interaction, passing by quickly, touching, tapping, pausing and staring, etc. To avoid unchanging interaction behavior, a large number of pedestrian pattern recognition parameters are introduced into the interaction design of the role-playing robot, including the distance of pedestrians stopping, time, intensity of interaction behavior, etc., to improve the degree of pattern recognition.

[0113] This embodiment is merely one feasible method for pedestrian status determination. It can be adaptively adjusted based on the same principle to suit different application scenarios. Gestures and voice information can also be used in behavior pattern discrimination to achieve deeper levels of interaction.

[0114] Based on the above embodiments, as an optional embodiment, the priority of the behavior pattern is recorded in the behavior pattern priority sequence;

[0115] Determining the emotional feedback pattern of the robot based on the behavioral pattern includes:

[0116] If the behavior mode is not the multi-person interaction mode, the emotional feedback mode associated with the behavior mode is searched from the pre-stored behavior mode-emotional feedback mode lookup table, and the found emotional feedback mode is used as the emotional feedback mode of the character robot.

[0117] When the behavior pattern is the multi-person interaction pattern, the behavior pattern of each pedestrian participating in the interaction is refined, and the behavior pattern with the highest priority is selected.

[0118] The system searches for the emotional feedback pattern associated with the highest priority behavior pattern in the pre-stored behavior pattern-emotional feedback pattern lookup table, and uses the found emotional feedback pattern as the emotional feedback pattern of the robot.

[0119] This invention prioritizes behavioral patterns to represent the strength of interactions corresponding to those patterns. Based on this strength, when a robot interacts with multiple people, strong interactive behaviors can draw the robot's attention away from weaker ones. This empowers the robot to communicate with multiple people simultaneously and encourages deeper interactions between pedestrians and the robot.

[0120] Based on the above embodiments, as an optional embodiment, determining the feedback action corresponding to the emotional feedback pattern includes:

[0121] Find the corresponding feedback action for the emotional feedback pattern from the pre-stored emotional feedback pattern-feedback action lookup table;

[0122] The step of obtaining the action parameters corresponding to the feedback action includes:

[0123] The motion parameters corresponding to the feedback motion are retrieved from the pre-stored motion library; the motion library stores each feedback motion designed for the robot character and its corresponding motion parameters.

[0124] It is understandable that the emotional feedback mode-feedback action reference table and each feedback action of the character robot and its corresponding action parameters are designed by the engineering designers during the design of the character robot's interaction scenario.

[0125] The emotional feedback modes include: boredom mode, fright mode, curiosity mode, clinginess mode, and excitement mode;

[0126] The feedback action is a combination of interactive actions of different body parts of the character robot.

[0127] When responding to pedestrian interactions, the robot's reactions should be guided by its overall personality profile. Emotional factors should be considered when designing feedback behaviors, and appropriate feedback behaviors should be designed for different emotional models of the robot.

[0128] Table 1 shows the interactive feedback design for a quadrupedal role robot.

[0129] Table 1 shows the feedback actions corresponding to each emotional feedback pattern and the emotional feedback actions corresponding to each behavioral pattern.

[0130] Table 1

[0131]

[0132] Based on the above embodiments, as an optional embodiment, the process of generating the action parameters corresponding to each feedback action includes:

[0133] Determine the interactive actions included in each of the feedback actions;

[0134] The motion parameters corresponding to each interactive action in each type of feedback action are generated using 3D modeling technology and animation retargeting technology.

[0135] Interpolation processing is performed on the action parameters corresponding to each interactive action to enable the robot to execute the corresponding interactive action smoothly and fluently.

[0136] Based on the action parameters corresponding to each interactive action after interpolation, the interactive actions in each feedback action are arranged to obtain the action parameters corresponding to each feedback action.

[0137] In the motion simulation system, verify the motion parameters corresponding to each feedback action. If the verification result shows that the control command generated according to the motion parameters corresponding to each feedback action can control the robot to execute each feedback action smoothly, output the motion parameters corresponding to each feedback action. If the verification result shows that the control command generated according to the motion parameters corresponding to each feedback action cannot control the robot to execute each feedback action smoothly, repeat the above operation.

[0138] The design and implementation of motion are core functions of interactive and performative role-playing robots. Therefore, vivid motion generation, smooth motion execution, and simple and convenient motion choreography are extremely important. Commercially available industrial robots or quadruped robots generally have relatively simple and unchanging motions, and only need to consider parameters such as motor operation time, angle, and speed; these motion parameters can be obtained simply through programming. However, for role-playing robots, the robots have more degrees of freedom and more coupled motions. Generating motion parameters through pure programming is not only tedious and time-consuming, but also results in stiff and unnatural motion execution.

[0139] The process of generating character robot motion is divided into four stages;

[0140] Action generation stage: Considering that the creation of character robot actions is more similar to the creation of animation, the UE4 game engine and the 3D software Blender were used to perform 3D modeling and animation redirection of different types of character robots to obtain action parameters that can be performed on the character robots.

[0141] Motion Smoothing Stage: After the robot's motion data is generated, the robot system needs to execute the motion smoothly. This involves trajectory planning and motion smoothing in robotics. In general robotics problems, this involves a trade-off between precise position control and smooth motion execution, making it a challenging problem. However, for a character robot, precise control of parameters such as position and velocity is less important; what's more important is the smoothness and naturalness of the overall motion. Therefore, this invention focuses on using discrete control to perform PVT (position-velocity-time polynomial interpolation) or PTP (point-to-point linear interpolation) interpolation on the robot's motion data to obtain a relatively smooth overall motion plan.

[0142] Motion choreography stage: The large amount of pre-stored motion data of the character robot needs to be properly integrated and scheduled to form a complete interaction process. Therefore, motion choreography is required for the previously generated motion data. This invention utilizes the animation system of the open-source software Blender to realize the motion pre-storage, motion fusion, and motion choreography of the character robot.

[0143] Virtual Simulation and Real-Time Mapping Stage: Robot simulation is a crucial step in verifying the robot's final operational performance. However, existing robot simulation software is often based on fixed robot model patterns, lacking freedom and placing high demands on the robot's underlying system for communication. This makes it difficult for non-robotics professionals to quickly learn and is not user-friendly for designers. Game engines, on the other hand, are sufficient to meet the design needs of character robots in terms of animation production, physics simulation, and rich interfaces. Therefore, this invention proposes to build a motion simulation system based on the UE4 engine that enables real-time communication and motion mapping. On one hand, UE4 has convenient interfaces that can accept both real-time BVH data from motion capture and animation skeletal data from animation software. On the other hand, UE4 has excellent graphical programming blueprints that facilitate communication with the robot to achieve real-time simulation and transmission of interactive actions.

[0144] By using the UE4 platform, robot motion capture or animation editing data can be matched with robot joint data using an animation skeleton remapping algorithm. This allows for the real-time generation of robot joint angle data, which is then uploaded to the robot for execution via communication. This process significantly simplifies traditional robot simulation and execution workflows, making it easier for robot operators to perform debugging tasks.

[0145] Based on the above embodiments, as an optional embodiment, the step of generating the motion parameters corresponding to each interactive action in each feedback action using 3D modeling technology and animation retargeting technology includes:

[0146] Based on 3D modeling technology, a robot animation model with a virtual animated skeleton is generated according to the number of joints, position, and degrees of freedom of the robot character.

[0147] The motion animation corresponding to each interactive action is generated using motion capture or animation software editing methods.

[0148] Based on the motion animation corresponding to each interactive action and the robot animation model, animation retargeting technology is used to generate motion parameters corresponding to each interactive action that can be executed on the character robot.

[0149] Figure 2 This example illustrates how action parameters are generated for interactive actions. ROS2 in the diagram is an open-source robot operating system. Figure 2 As shown, in the 3D modeling of this invention, a robot animation model with a virtual animation skeleton needs to be generated based on the number of joints, position, and degrees of freedom of the robot itself. Then, the robot model's motion animation is generated by motion capture or by the animator manually dragging and arranging. The angle data of each degree of freedom of the robot model is extracted from the game engine. After motion mapping and correction, the robot's executable motion data can be obtained, waiting for the robot to call it.

[0150] Secondly, the present invention describes the human-computer interaction device for a role-playing robot, and the human-computer interaction device for a role-playing robot described below and the human-computer interaction method for a role-playing robot described above can be referred to in correspondence with each other. Figure 3 An example is a schematic diagram of a human-computer interaction device for a role-playing robot, the device comprising:

[0151] Acquisition module 21 is used to acquire multi-source information within the target area;

[0152] The behavior pattern determination module 22 is used to determine the behavior pattern of pedestrians in the target area based on the multi-source information.

[0153] The emotional feedback pattern determination module 23 is used to determine the emotional feedback pattern of the role robot based on the behavior pattern.

[0154] The interaction module 24 is used to determine the feedback action corresponding to the emotional feedback mode, obtain the action parameters corresponding to the feedback action, and generate control instructions according to the action parameters. The control instructions are used to control the character robot to perform the feedback action.

[0155] The human-computer interaction device for role-playing robots provided in this embodiment of the invention specifically executes the processes of the above-described human-computer interaction method embodiments for role-playing robots. For details, please refer to the contents of the above-described human-computer interaction method embodiments for role-playing robots, which will not be repeated here.

[0156] This invention provides a human-computer interaction device for a role-playing robot, which fuses multi-source information to determine the current interaction subject's behavior; and invokes corresponding feedback actions based on the interaction subject's behavior to make the role-playing robot's interaction response more appropriate, thereby improving the role-playing robot's interaction effect.

[0157] Thirdly, Figure 4 An example is a schematic diagram of the physical structure of an electronic device. For example... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a human-computer interaction method for a role-playing robot. This method includes: acquiring multi-source information within a target area; determining the behavior pattern of pedestrians within the target area based on the multi-source information; determining the emotional feedback pattern of the role-playing robot based on the behavior pattern; determining the feedback action corresponding to the emotional feedback pattern, acquiring the action parameters corresponding to the feedback action, and generating control instructions according to the action parameters. The control instructions are used to control the role-playing robot to execute the feedback action.

[0158] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] Fourthly, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, it performs a human-computer interaction method for a role-playing robot, the method comprising: acquiring multi-source information within a target area; determining a pedestrian behavior pattern within the target area based on the multi-source information; determining an emotional feedback pattern of the role-playing robot based on the behavior pattern; determining a feedback action corresponding to the emotional feedback pattern, acquiring action parameters corresponding to the feedback action, and generating a control instruction according to the action parameters, the control instruction being used to control the role-playing robot to perform the feedback action.

[0160] Fifthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon for executing a human-computer interaction method for a role-playing robot. The method includes: acquiring multi-source information within a target area; determining a pedestrian behavior pattern within the target area based on the multi-source information; determining an emotional feedback pattern of the role-playing robot based on the behavior pattern; determining a feedback action corresponding to the emotional feedback pattern and acquiring action parameters corresponding to the feedback action; generating a control instruction according to the action parameters; the control instruction being used to control the role-playing robot to execute the feedback action.

[0161] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A human-computer interaction method for a role-playing robot, characterized in that, The method includes: Acquire multi-source information within the target area; Based on the multi-source information, the behavioral patterns of pedestrians within the target area are determined; the behavioral patterns include unmanned interaction mode, passing mode, multi-person interaction mode, touch mode, tapping mode, and staring mode. Based on the behavioral pattern, the emotional feedback pattern of the robot is determined; The feedback action corresponding to the emotional feedback mode is determined, and the action parameters corresponding to the feedback action are obtained. A control command is generated according to the action parameters. The control command is used to control the character robot to execute the feedback action. The character robot has set character attributes. The feedback action is a combination of interactive actions of different body parts of the character robot, guided by the overall personality setting of the character robot. The priority of the behavior pattern is recorded in the behavior pattern priority sequence; Determining the emotional feedback pattern of the robot based on the behavioral pattern includes: When the behavior pattern is a multi-person interaction mode, the behavior pattern of each pedestrian participating in the interaction is refined, and the behavior pattern with the highest priority is selected. The system searches for the emotional feedback pattern associated with the highest priority behavior pattern in the pre-stored behavior pattern-emotional feedback pattern lookup table, and uses the found emotional feedback pattern as the emotional feedback pattern of the robot.

2. The human-computer interaction method for a role-playing robot according to claim 1, characterized in that, Acquire multi-source information within the target area, specifically including: Multi-source information within the target area is collected using multi-source sensors.

3. The human-computer interaction method for a role-playing robot according to claim 2, characterized in that, The multi-source information within the target area is collected using multi-source sensors, specifically including at least two of the following: The multi-source sensor includes an image sensor, which is used to acquire scene images of the target area; the target area is a region within a certain range surrounding the robot. The multi-source sensor includes a distance sensor, which is used to collect distance information between the robot and pedestrians within the target area. The multi-source sensor includes a pressure sensor, which is used to collect pressure information of the robot. The multi-source sensor includes a vibration sensor, which utilizes the vibration information of the robot.

4. The human-computer interaction method for a role-playing robot according to claim 3, characterized in that, The scene image of the target area, the distance information between the robot and the pedestrian in the target area, the pressure information of the robot, and the vibration information of the robot are used as the multi-source information. Determining the pedestrian behavior pattern within the target area based on the multi-source information includes: Perform facial recognition on the scene image of the target area; If the robot's pressure / vibration information indicates that it is under pressure / vibration, then the touch / tapping Boolean value will be recorded as positive; otherwise, the touch / tapping Boolean value will not be recorded as positive. Based on the facial recognition results, the distance information between the robot and the pedestrian in the target area, the touch Boolean value, and the tap Boolean value, the behavior pattern of the pedestrian in the target area is determined.

5. The human-computer interaction method for a role-playing robot according to claim 4, characterized in that, The process of determining the pedestrian's behavior pattern within the target area based on facial recognition results, distance information between the robot and the pedestrian within the target area, touch Boolean values, and tapping Boolean values ​​includes: If the face recognition result is that no human figure image envelope is detected, then the behavior mode is the unmanned interaction mode; If the human image recognition result is that at least one human image envelope is detected and all detected human image envelopes meet the preset conditions, then the behavior mode is the pass-through mode. If the human image recognition result is that multiple human image envelopes are identified and at least two of the identified human image envelopes do not meet the preset conditions, then the behavior mode is a multi-person interaction mode. If the human image recognition result is that a human figure image envelope is detected, the detected human figure image envelope does not meet the preset conditions, and the touch Boolean value is positive, then the behavior mode is touch mode; If the human image recognition result is that a human figure image envelope is detected, the detected human figure image envelope does not meet the preset conditions, and the tapping Boolean value is positive, then the behavior mode is the tapping mode. If the human image recognition result is that a human figure image envelope is detected, the detected human figure image envelope does not meet the preset conditions, the spatial position of the detected human figure image envelope only changes within the preset spatial range, the touch Boolean value is not positive and the tap Boolean value is not positive, then the behavior mode is the gaze mode. The variation in the spatial position of the envelope of the identified human figure image is determined by the distance information between the robot and the pedestrian within the target area; The preset condition is that the time it takes for the human-shaped image envelope to pass through the robot is less than a preset time.

6. The human-computer interaction method for a role-playing robot according to claim 5, characterized in that, Determining the emotional feedback pattern of the robot based on the behavioral pattern includes: If the behavior pattern is not the multi-person interaction pattern, the emotional feedback pattern associated with the behavior pattern is searched from the pre-stored behavior pattern-emotional feedback pattern lookup table, and the found emotional feedback pattern is used as the emotional feedback pattern of the character robot.

7. The human-computer interaction method for a role-playing robot according to claim 1, characterized in that, The step of determining the feedback action corresponding to the emotional feedback pattern includes: Find the corresponding feedback action for the emotional feedback pattern from the pre-stored emotional feedback pattern-feedback action lookup table; The step of obtaining the action parameters corresponding to the feedback action includes: The motion parameters corresponding to the feedback motion are retrieved from the pre-stored motion library; the motion library stores each feedback motion designed for the robot character and its corresponding motion parameters. The emotional feedback modes include: boredom mode, fright mode, curiosity mode, clinginess mode, and excitement mode; The feedback action is a combination of interactive actions of different body parts of the character robot.

8. The human-computer interaction method for a role-playing robot according to claim 7, characterized in that, The process of generating the action parameters corresponding to each feedback action includes: Determine the interactive actions included in each of the feedback actions; The motion parameters corresponding to each interactive action in each type of feedback action are generated using 3D modeling technology and animation retargeting technology. Interpolation processing is performed on the action parameters corresponding to each interactive action to enable the robot to execute the corresponding interactive action smoothly and fluently. Based on the action parameters corresponding to each interactive action after interpolation, the interactive actions in each feedback action are arranged to obtain the action parameters corresponding to each feedback action. In the motion simulation system, verify the motion parameters corresponding to each feedback action. If the verification result shows that the control command generated according to the motion parameters corresponding to each feedback action can control the robot to execute each feedback action smoothly, output the motion parameters corresponding to each feedback action. If the verification result shows that the control command generated according to the motion parameters corresponding to each feedback action cannot control the robot to execute each feedback action smoothly, repeat the above operation.

9. The human-computer interaction method for a role-playing robot according to claim 8, characterized in that, The method of generating motion parameters corresponding to each interactive action in each feedback action using 3D modeling technology and animation retargeting technology includes: Based on 3D modeling technology, a robot animation model with a virtual animated skeleton is generated according to the number of joints, position, and degrees of freedom of the robot character. The motion animation corresponding to each interactive action is generated using motion capture or animation software editing methods. Based on the motion animation corresponding to each interactive action and the robot animation model, animation retargeting technology is used to generate motion parameters corresponding to each interactive action that can be executed on the character robot.

10. A human-computer interaction device for a role-playing robot, characterized in that, The device includes: The acquisition module is used to acquire multi-source information within the target area; The behavior pattern determination module is used to determine the behavior patterns of pedestrians in the target area based on the multi-source information; the behavior patterns include unmanned interaction mode, passing mode, multi-person interaction mode, touch mode, tapping mode, and staring mode. An emotion feedback pattern determination module is used to determine the emotion feedback pattern of the role robot based on the behavior pattern. An interaction module is used to determine the feedback action corresponding to the emotional feedback mode, obtain the action parameters corresponding to the feedback action, and generate control instructions according to the action parameters. The control instructions are used to control the character robot to execute the feedback action. The character robot has set character attributes, and the feedback action is a combination of interactive actions of different body parts of the character robot, guided by the overall personality setting of the character robot. The priority of the behavior pattern is recorded in the behavior pattern priority sequence; The emotional feedback pattern determination module is specifically used for: When the behavior pattern is a multi-person interaction mode, the behavior pattern of each pedestrian participating in the interaction is refined, and the behavior pattern with the highest priority is selected. The system searches for the emotional feedback pattern associated with the highest priority behavior pattern in the pre-stored behavior pattern-emotional feedback pattern lookup table, and uses the found emotional feedback pattern as the emotional feedback pattern of the robot.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the human-computer interaction method for a role-playing robot as described in any one of claims 1 to 10.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the human-computer interaction method for a role-playing robot as described in any one of claims 1 to 10.

Images