A robot interaction method, device, storage medium and equipment

Abstract

The present application discloses a robot interaction method, device, storage medium and equipment, the method comprising: first obtaining a face image of a target user, then using the face image of the target user to determine the sight angle of the target user; then determining the terminal position information of the target robot's movement based on the face image and sight angle of the target user; and then controlling the target robot to move to the terminal position. It can be seen that since the present application determines the terminal position of the robot's movement based on the user's sight movement, and then controls the robot to move to the terminal position, it can achieve more accurate and convenient robot interaction, thereby improving the interaction effect and the user's interaction experience.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robot interaction method, device, storage medium and equipment. Background Art

[0002] With the advancement of robotics technology, the types of robots are becoming more and more diverse, and their functions are becoming more and more powerful. Among them, the use of interactive robots is becoming more and more extensive. Through interaction with robots, people's lives can be greatly facilitated and comfortable.

[0003] At present, the traditional robot interaction method is that the user controls the robot based on external sensors or human limb movements. Among them, when the robot is interacted with based on external sensors, additional equipment is required, which is costly and has poor scalability. The wearer of the equipment needs to make prescribed standard movements to correctly guide the movement of the robot; and when the robot is interacted with based on human limb movements, due to the high uncertainty of the human body and the large degree of variability of limb movements, the algorithm accuracy is not high, making the robot movement and positioning inaccurate. Therefore, these two robot interaction methods are inconvenient to use and the control accuracy is unstable, which will lead to poor interaction effects and affect the user's interaction experience with the robot.

[0004] Therefore, how to achieve more accurate and convenient robot interaction to improve the interaction effect and user interaction experience is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to provide a robot interaction method, device, storage medium and equipment, which can accurately determine the end point of the robot's movement according to the user's line of sight movement, facilitate the robot to move accurately, thereby achieving more accurate and convenient robot interaction, and further improving the interaction effect and the user's interaction experience.

[0006] The present application embodiment provides a robot interaction method, including:

[0007] Obtain the target user's facial image;

[0008] Determine the sight angle of the target user by using the face image of the target user;

[0009] Determining the terminal position information of the target robot according to the face image of the target user and the sight angle of the target user;

[0010] Control the target robot to move to the end position.

[0011] In a possible implementation, before acquiring the face image of the target user, the method further includes:

[0012] Receive a preset start instruction, and control the target robot to enter an interactive mode according to the preset start instruction.

[0013] In a possible implementation, the preset start instruction includes a preset voice start instruction; and receiving the preset start instruction and controlling the target robot to enter the interactive mode according to the preset start instruction includes:

[0014] Receive a preset voice start command, and determine the location information of a target user interacting with the target robot according to the microphone array to which the voice belongs;

[0015] According to the position information of the target user, the target robot is controlled to rotate so that the face of the target user appears in the shooting device of the target robot, so as to interact with the target user.

[0016] In a possible implementation, obtaining the face image of the target user includes:

[0017] According to the position information of the target user and the current position information of the target robot, the target robot is controlled to rotate so that the distance between the target robot and the target user is within a preset range;

[0018] The facial image of the target user is captured using the shooting device of the target robot.

[0019] In a possible implementation, the determining the sight angle of the target user by using the face image of the target user includes:

[0020] Inputting the face image of the target user into a pre-built sight line angle recognition model to identify the sight line angle of the target user;

[0021] Among them, the sight angle recognition model includes a backbone network and a fully connected layer; the sight angle includes a vertical angle and a horizontal angle.

[0022] In a possible implementation, the sight angle recognition model is constructed as follows:

[0023] Obtaining facial images of sample users;

[0024] Inputting the facial image of the sample user into an initial sight angle recognition model, and performing training using a preset objective function to generate the sight angle recognition model;

[0025] Among them, the initial sight angle recognition model includes an initial backbone network and an initial fully connected layer.

[0026] In a possible implementation manner, the preset objective function is a mean square error function MSE.

[0027] In a possible implementation, inputting the face image of the target user into a pre-built sight line angle recognition model to identify the sight line angle of the target user includes:

[0028] Inputting the face image of the target user into the backbone network of the sight angle recognition model to obtain a backbone feature vector of the face image of the target user;

[0029] The backbone feature vector of the target user's facial image is input into the fully connected layer of the sight line angle recognition model for angle regression processing to obtain the vertical angle and horizontal angle of the target user's sight line.

[0030] In a possible implementation, determining the terminal position information of the target robot according to the face image of the target user and the sight angle of the target user includes:

[0031] Determine the face center position information of the target user according to the face image of the target user;

[0032] The terminal position information of the target robot is calculated according to the face center position information of the target user and the sight angle of the target user.

[0033] The present application also provides a robot interaction device, the device comprising:

[0034] A first acquisition unit, used to acquire a face image of a target user;

[0035] A first determining unit, configured to determine a sight angle of the target user using a face image of the target user;

[0036] A second determining unit is used to determine the terminal position information of the target robot according to the face image of the target user and the sight angle of the target user;

[0037] The first control unit is used to control the target robot to move to the end position.

[0038] In a possible implementation manner, the device further includes:

[0039] The second control unit is used to receive a preset start instruction and control the target robot to enter an interactive mode according to the preset start instruction.

[0040] In a possible implementation, the preset start instruction includes a preset voice start instruction; and the second control unit includes:

[0041] A first determination subunit is used to receive a preset voice start instruction and determine the position information of a target user interacting with the target robot according to the microphone array to which the voice belongs;

[0042] The first control subunit is used to control the target robot to rotate according to the position information of the target user, so that the face of the target user appears in the shooting device of the target robot, so as to interact with the target user.

[0043] In a possible implementation manner, the first acquiring unit includes:

[0044] a second control subunit, configured to control the target robot to rotate according to the position information of the target user and the current position information of the target robot, so that the distance between the target robot and the target user is within a preset range;

[0045] The shooting subunit is used to use the shooting device of the target robot to obtain the facial image of the target user.

[0046] In a possible implementation manner, the first determining unit is specifically configured to:

[0047] Inputting the face image of the target user into a pre-built sight line angle recognition model to identify the sight line angle of the target user;

[0048] Among them, the sight angle recognition model includes a backbone network and a fully connected layer; the sight angle includes a vertical angle and a horizontal angle.

[0049] In a possible implementation manner, the device further includes:

[0050] A second acquisition unit, used to acquire a face image of a sample user;

[0051] A training unit, configured to input the facial image of the sample user into an initial sight angle recognition model, perform training using a preset objective function, and generate the sight angle recognition model;

[0052] Among them, the initial sight angle recognition model includes an initial backbone network and an initial fully connected layer.

[0053] In a possible implementation manner, the preset objective function is a mean square error function MSE.

[0054] In a possible implementation manner, the first determining unit includes:

[0055] A first input subunit, configured to input the face image of the target user into the backbone network of the sight angle recognition model to obtain a backbone feature vector of the face image of the target user;

[0056] The second input subunit is used to input the backbone feature vector of the target user's facial image into the fully connected layer of the sight line angle recognition model for angle regression processing to obtain the vertical direction angle and horizontal direction angle of the target user's sight line.

[0057] In a possible implementation manner, the second determining unit includes:

[0058] A second determining subunit is used to determine the face center point position information of the target user according to the face image of the target user;

[0059] The calculation subunit is used to calculate the terminal position information of the target robot according to the face center position information of the target user and the sight angle of the target user.

[0060] The embodiment of the present application also provides a robot interaction device, including: a processor, a memory, and a system bus;

[0061] The processor and the memory are connected via the system bus;

[0062] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementations of the above-mentioned robot interaction method.

[0063] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes any one of the implementations of the above-mentioned robot interaction method.

[0064] The embodiment of the present application also provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes any one of the implementation modes of the above-mentioned robot interaction method.

[0065] The robot interaction method, device, storage medium and equipment provided in the embodiment of the present application first obtain the face image of the target user, then use the face image of the target user to determine the sight angle of the target user; then, according to the face image of the target user and the sight angle of the target user, determine the terminal position information of the movement of the target robot; and then control the target robot to move to the terminal position. It can be seen that since the embodiment of the present application determines the terminal position of the robot movement according to the movement of the user's sight, and then controls the robot to move to the terminal position, it can achieve more accurate and convenient robot interaction, thereby improving the interaction effect and the user's interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 A schematic diagram of a flow chart of a robot interaction method provided in an embodiment of the present application;

[0068] Figure 2 A schematic diagram of a robot interaction scenario provided in an embodiment of the present application;

[0069] Figure 3 A schematic diagram of the sight direction angle provided in an embodiment of the present application;

[0070] Figure 4 A schematic diagram of a process for determining the sight angle of a target user using a face image of the target user provided in an embodiment of the present application;

[0071] Figure 5 A schematic diagram of the composition of a robot interaction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] With the rapid development of robotics technology, the use of interactive robots is becoming more and more widespread. Interaction with robots can bring great convenience and comfortable experience to people's lives. Existing robot interaction methods usually include the following two methods:

[0073] The first is a method of robot interaction based on wearable devices. This interaction method requires additional hardware sensors, and the corresponding specified actions are performed by wearing sensors to control the movement of the robot. For example, the user can control the robot to move accordingly by moving and clicking the bracelet, or by using the bracelet to emit laser lines. However, in real life, due to the influence of equipment and environment, the interaction accuracy of this method is greatly dependent on hardware equipment, and because additional equipment is required, the cost is high and the scalability is poor. The wearer of the device is also required to perform prescribed standard actions to correctly guide the movement of the robot. Not only is there inconvenience, but there is also instability in the control accuracy, resulting in poor interaction effects.

[0074] The second method is to interact with the robot based on human limb movements. This interaction method requires judging limb movements based on key point detection of the human body to control the robot. For example, the user can control the robot to move accordingly through gestures of the arm or finger. However, due to the high uncertainty of the human body and the large degree of variation in limb movements, the accuracy of the interaction algorithm is not high, the robot movement positioning is not accurate enough, and the interaction effect is poor.

[0075] Therefore, the above two robot interaction methods are inconvenient to use and unstable in control accuracy, which will lead to poor interaction effects and affect the interaction experience between users and robots.

[0076] In order to solve the above defects, the present application provides a robot interaction method, which first obtains the face image of the target user, and then uses the face image of the target user to determine the sight angle of the target user; then, according to the face image of the target user and the sight angle of the target user, the terminal position information of the target robot's movement is determined; and then the target robot can be controlled to move to the terminal position. It can be seen that since the embodiment of the present application determines the terminal position of the robot's movement based on the user's sight movement, and then controls the robot to move to the terminal position, it can achieve more accurate and convenient robot interaction, thereby improving the interaction effect and the user's interaction experience.

[0077] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0078] First embodiment

[0079] See also Figure 1, is a flow chart of a robot interaction method provided in this embodiment, the method comprising the following steps:

[0080] S101: Acquire a face image of a target user.

[0081] In this embodiment, any robot that uses the interaction method provided by this application for human-machine interaction is defined as a target robot, and the user who interacts with it is defined as a target user. It should be noted that this application does not limit the type of the target robot, for example, the target robot can be a service robot or an operating robot, and this application does not limit the application scenario of the target robot, for example, the target robot can be a household robot or a robot serving in a restaurant, etc.

[0082] Figure 2 FIG. 1 shows an exemplary scenario diagram of a robot interaction method that can be applied to an embodiment of the present application. Figure 2 As shown, the target robot 201 and the target user 202 are located in the same room. The target robot 201 can be any household robot, such as a sweeping robot, etc. The target user 202 can be any family member at home. It should be understood that Figure 2 The number and position of the target robots and target users in the figure are only for illustration purposes. Any number of target robots and target users may be provided as required.

[0083] The robot interaction method provided in the embodiment of the present application is generally executed by a controller in the target robot 201, and accordingly, the robot interaction device can be set in the target robot 201. However, it is easy for those skilled in the art to understand that the robot interaction method provided in the embodiment of the present application can also be executed by other terminal devices.

[0084] In order to achieve more accurate and convenient robot interaction, thereby improving the interaction effect and the user's interaction experience, firstly, it is necessary to obtain a face image of the target user to perform the subsequent step S102.

[0085] Specifically, an optional implementation method is that before obtaining the facial image of the target user, it is also necessary to receive a preset start-up instruction, and control the target robot to enter the interactive mode according to the preset start-up instruction, that is, wake up the target robot according to the preset start-up instruction. The content of the specific preset start-up instruction (that is, the wake-up method) is not limited. For example, voice, gestures or other devices (such as bracelets, etc.) can be used as preset start-up instructions to wake up the target robot. After waking up, the target robot can be controlled to stop the current work, enter the interactive mode, and wait for interaction with the target user.

[0086] Specifically, an optional implementation method is that when the preset start-up instruction includes a preset voice start-up instruction, such as after waking up the target robot by voice, first, the microphone array to which the voice belongs can be used to determine the location information of the target user interacting with the target robot, that is, the existing or future positioning method can be used in combination with the microphone array to locate the location information of the target user, and then, in order to achieve more accurate and convenient robot interaction to improve the interaction effect and user interaction experience, it is further possible to obtain the target user's face image based on the target user's location information using existing or future image acquisition methods to execute subsequent steps S103 and S104.

[0087] Specifically, after determining the location information of the target user, if it is determined that the target user does not appear in the shooting device (such as lens, etc.) of the target robot, then in order to achieve convenient and accurate interaction between the target user and the target robot, the target robot can be further controlled to rotate according to the location information of the target user so that the face of the target user can appear in the shooting device (such as lens) of the target robot, thereby facilitating the interaction between the target robot and the target user.

[0088] Further, the target robot's shooting device (such as a camera) can be used to shoot the overall image of the target user, and the overall image can be used to perform human body segmentation. At the same time, after determining the human body depth information of the target user using the depth map information obtained by the depth sensor, the distance between the target user and the target robot can be determined. In order to more accurately obtain the target user's line of sight movement information, if it is determined that the distance between the target user and the target robot is far, it is necessary to control the target robot to rotate so that the distance between the target robot and the target user is within a preset range, and then the target robot's shooting device can be used to shoot the target user's facial image, and ensure that the captured target user's facial image is clear enough, so that the facial image can be used to achieve more accurate and convenient robot interaction through subsequent steps S102-S104.

[0089] Among them, the specific value of the preset range can be determined according to the actual situation and experience value, and the embodiment of the present application does not limit this. For example, the preset range can be set to within 3 meters.

[0090] Furthermore, it should be noted that the present embodiment does not limit the types of the overall image of the target user and the facial image of the target user. For example, the overall image of the target user and the facial image of the target user can both be color images composed of the three primary colors of red (R), green (G), and blue (B), or can be grayscale images, etc.

[0091] In addition, this embodiment does not limit the resolution of the target user's overall image and the target user's facial image. For example, the target user's overall image can be a 720*480 RGB image, and the target user's facial image can be a high-resolution 1920*1280 RGB image.

[0092] S102: Determine the sight angle of the target user by using the face image of the target user.

[0093] In this embodiment, after the facial image of the target user is acquired through step S101, in order to enable convenient and accurate interaction between the target user and the target robot, it is further necessary to utilize existing or future image recognition technology to identify the facial image of the target user, so as to determine the sight angle of the target user based on the recognition result.

[0094] Specifically, an optional implementation method is to input the target user's facial image into a pre-built sight angle recognition model to identify the target user's sight angle. The sight angle recognition model includes a backbone network and a fully connected layer. The sight angle includes a vertical angle and a horizontal angle.

[0095] In this implementation, the target user's gaze angle can be identified by a pre-built gaze angle recognition model. For the estimation of three-dimensional gaze, the target user's gaze direction is derived from the target user's face reminder. It should be noted that in the camera coordinate system, the target user's gaze direction depends not only on the state of the eyes (eyeball position, eye opening degree, etc.), but also on the head posture. Usually, the target user's gaze direction is determined by the vertical direction (such as Figure 3 α as shown) and horizontal direction (as Figure 3 The two angles θ) are shown.

[0096] It should be noted that, since the sight angle of the target user is jointly determined by the sight direction (including the vertical direction α and the horizontal direction θ) and the head orientation, after obtaining the face image of the target user (such as a high-resolution 1920*1280RGB image), it can be input into the backbone network (such as Resnet backbone) of the sight angle recognition model to obtain the backbone feature vector of the face image of the target user. Then, the backbone feature vector can be input into the fully connected layer (Fully connected layer, referred to as FC layer) of the sight angle recognition model for angle regression processing to obtain the vertical direction angle (here defined as α') and horizontal direction angle (here defined as θ') of the sight of the target user, such as Figure 4 As shown, it is used to execute the subsequent step S103.

[0097] Next, this embodiment will introduce the construction process of the sight angle recognition model, which may specifically include the following steps A1-A2:

[0098] Step A1: Obtain a face image of a sample user.

[0099] In this embodiment, in order to build a line of sight angle recognition model, a lot of preparatory work needs to be done in advance. First, it is necessary to collect facial images of a large number of users as facial images of sample users. For example, 100 facial images of different users can be collected in advance, and each facial image of the collected different users can be used as a facial image of a sample user. The sample line of sight angles (including vertical angles and horizontal angles) corresponding to the facial images of these sample users are manually annotated in advance to train the line of sight angle recognition model.

[0100] Step A2: Input the facial image of the sample user into the initial sight angle recognition model, train it using a preset objective function, and generate a sight angle recognition model.

[0101] In this embodiment, after obtaining the facial image of the sample user through step A1, further, a facial image of the sample user can be sequentially input into the initial sight angle recognition model to obtain the sight angle recognition result of the sample user, and the recognition result is compared with the corresponding manual annotation result, and according to the difference between the two, and the preset objective function (which can be set according to the actual situation and experience value, and is not limited in this application), multiple rounds of model training are performed until the training end condition is met. At this time, the sight angle recognition model can be generated. Among them, the initial sight angle recognition model may include an initial backbone network and an initial fully connected layer. However, the specific composition of the initial sight angle recognition model is not limited. For example, the initial sight angle recognition model can be a deep neural network model such as a residual network (Residual Network, abbreviated as ResNet) or a recurrent neural network (Recurrent Neural Network, abbreviated as RNN).

[0102] During the training process, an optional implementation method is that the preset objective function can use the mean squared error function (MSE). Then, the MSE-loss can be calculated by GroundTruth, and the model parameters can be updated by the Adam parameter optimizer. That is, during the model training process, the model parameters of the sight angle recognition model can be continuously updated according to the change of the MSE-loss value until the MSE-loss value meets the requirements. For example, if the change is very small, the update of the model parameters is stopped, and the training of the sight angle recognition model is completed to obtain the trained sight angle recognition model.

[0103] Through the above embodiment, the sight angle recognition model can be generated by training the face image of the sample user. Further, the generated sight angle recognition model can be verified by using the face image of the verification user. The specific verification process may include the following steps B1-B3:

[0104] Step B1: Obtain a facial image of the user to be verified.

[0105] In actual applications, in order to verify the line of sight angle recognition model, it is first necessary to obtain the face image of the verification user, where the face image of the verification user refers to a face image that can be used to verify the line of sight angle recognition model. After obtaining the face image of the verification user, step B2 can be continued.

[0106] Step B2: Input the face image of the verification user into the sight angle recognition model to obtain the sight angle of the verification user.

[0107] In the specific implementation process, after obtaining the facial image of the verified user through step B1, the facial image of the verified user can be further input into the line of sight angle recognition model to obtain the line of sight angle (including vertical angle and horizontal angle) of the verified user, and then step B3 can be continued.

[0108] Step B3: When the verification user's sight angle (including vertical angle and horizontal angle) recognition result is inconsistent with the verification user's sight angle marking result corresponding to the verification user's facial image, the verification user's facial image is used again as the sample user's facial image, and the sight angle recognition model is updated.

[0109] Through the above embodiments, the line of sight angle recognition model can be effectively verified by using the facial image of the verified user. When the recognition result of the line of sight angle of the verified user is inconsistent with the corresponding manual annotation result of the line of sight angle of the verified user (or the error is not within the preset angle range), the line of sight angle recognition model can be adjusted and updated in time, which helps to improve the recognition precision and accuracy of the recognition model.

[0110] S103: Determine the terminal position information of the target robot according to the face image of the target user and the sight angle of the target user.

[0111] In this embodiment, after acquiring the facial image of the target user through step S101 and determining the sight angle of the target user through step S102, the facial image of the target user and the sight angle of the target user can be further analyzed and processed to determine the terminal position information of the target robot's movement based on the processing results, so as to execute the subsequent step S104.

[0112] Specifically, an optional implementation method is to first determine the target user's face center position information based on the target user's face image. Specifically, in this implementation method, the target user's face image and the target user's body depth information can be processed to determine the target user's face center position as the body position, and its coordinates are defined as (x p ,y p ,z p ).

[0113] Then, combined with the sight angle of the target user obtained in step S103 (including the vertical angle α' and the horizontal angle θ'), the terminal position information of the target robot can be calculated by the following calculation formula:

[0114]

[0115] Among them, (x g ,y g ,z g ) represents the coordinate information of the end point position of the target robot movement.

[0116] It should be noted that (x g ,y g ,z g ) is the target user's gaze point on the ground. Specifically, the target user's line of sight can be regarded as a ray directed to the ground, and then the intersection of the ray and the ground plane can be used as the target user's gaze point on the ground. And the gaze point on the ground is also the spatial end point to which the robot needs to move in the camera coordinate system. It should also be noted that the camera coordinate system can also be converted into a robot coordinate system with the center point of the robot (torso) as the zero point according to actual needs, and this application does not limit this.

[0117] It should be noted that after regarding the target user's line of sight as a ray directed toward the ground, when the intersection of the ray and the ground plane (i.e., the position of the target user's gaze point on the ground) exceeds the room where the target robot and the target user are located, the intersection of the ray and the ground plane outside the room (i.e., the position of the target user's gaze point on the ground) and the connection line of the current position of the target robot and the wall of the room can be used as the end point of the target robot's movement, that is, the target robot can be controlled to move to the side of the room wall through the subsequent step S104, as close as possible to the position of the intersection with the room wall.

[0118] S104: Control the target robot to move to the end position.

[0119] In this embodiment, after determining the destination position information of the target robot's movement through step S103, the movement route can be further planned in combination with the target robot's current position information, the destination position information, and the grid map to avoid obstacles on the route, and then the target robot can be controlled to move to the destination position according to the movement route, thereby achieving more accurate and convenient robot interaction, thereby improving the interaction effect and the user's interaction experience.

[0120] Furthermore, when the target robot moves to the final position, it can also receive various command information issued by the target user, such as voice, gesture and other command information, and perform corresponding operations according to the analysis results obtained by analyzing the command information, such as temperature measurement, charging, picking up objects, cleaning, etc.

[0121] For example: when the target robot is a sweeping robot, after it moves to the final position, it can receive cleaning instructions issued by the target user through voice, gestures or other devices (such as mobile client, etc.), and determine the cleaning range according to the cleaning instruction for cleaning. For example, the final position where the sweeping robot moves to can be used as the center of the circle, and the circular area defined by the preset radius (or the radius specified by the user through the cleaning instruction) can be used as the cleaning range for cleaning; or the target user can also use the mobile client to define a custom cleaning range (such as a rectangle, circle or other irregular polygon, etc.) near the final position where the sweeping robot moves to, and send the custom cleaning range to the sweeping robot through the cleaning instruction, so that the sweeping robot can clean according to the custom cleaning range.

[0122] In summary, the robot interaction method provided in this embodiment first obtains the face image of the target user, then uses the face image of the target user to determine the sight angle of the target user; then, according to the face image of the target user and the sight angle of the target user, determines the terminal position information of the movement of the target robot; and then controls the target robot to move to the terminal position. It can be seen that since the embodiment of the present application determines the terminal position of the robot movement according to the movement of the user's sight, and then controls the robot to move to the terminal position, it can achieve more accurate and convenient robot interaction, thereby improving the interaction effect and the user's interaction experience.

[0123] Second embodiment

[0124] This embodiment will introduce a robot interaction device. For related content, please refer to the above method embodiment.

[0125] See also Figure 5 , is a schematic diagram of the composition of a robot interaction device provided in this embodiment, the device 500 includes:

[0126] The first acquisition unit 501 is used to acquire a face image of a target user;

[0127] A first determining unit 502 is used to determine the sight angle of the target user by using the face image of the target user;

[0128] A second determining unit 503 is used to determine the terminal position information of the target robot according to the face image of the target user and the sight angle of the target user;

[0129] The first control unit 504 is used to control the target robot to move to the end position.

[0130] In an implementation of this embodiment, the device further includes:

[0131] The second control unit is used to receive a preset start instruction and control the target robot to enter an interactive mode according to the preset start instruction.

[0132] In an implementation of this embodiment, the preset start instruction includes a preset voice start instruction; and the second control unit includes:

[0133] A first determination subunit is used to receive a preset voice start instruction and determine the position information of a target user interacting with the target robot according to the microphone array to which the voice belongs;

[0134] The first control subunit is used to control the target robot to rotate according to the position information of the target user, so that the face of the target user appears in the shooting device of the target robot, so as to interact with the target user.

[0135] In an implementation of this embodiment, the first acquiring unit 501 includes:

[0136] a second control subunit, configured to control the target robot to rotate according to the position information of the target user and the current position information of the target robot, so that the distance between the target robot and the target user is within a preset range;

[0137] The shooting subunit is used to use the shooting device of the target robot to obtain the facial image of the target user.

[0138] In an implementation of this embodiment, the first determining unit 502 is specifically configured to:

[0139] Inputting the face image of the target user into a pre-built sight line angle recognition model to identify the sight line angle of the target user;

[0140] Among them, the sight angle recognition model includes a backbone network and a fully connected layer; the sight angle includes a vertical angle and a horizontal angle.

[0141] In an implementation of this embodiment, the device further includes:

[0142] A second acquisition unit, used to acquire a face image of a sample user;

[0143] A training unit, configured to input the facial image of the sample user into an initial sight angle recognition model, perform training using a preset objective function, and generate the sight angle recognition model;

[0144] Among them, the initial sight angle recognition model includes an initial backbone network and an initial fully connected layer.

[0145] In an implementation manner of this embodiment, the preset objective function is a mean square error function MSE.

[0146] In an implementation of this embodiment, the first determining unit 502 includes:

[0147] A first input subunit, configured to input the face image of the target user into the backbone network of the sight angle recognition model to obtain a backbone feature vector of the face image of the target user;

[0148] The second input subunit is used to input the backbone feature vector of the target user's facial image into the fully connected layer of the sight line angle recognition model for angle regression processing to obtain the vertical direction angle and horizontal direction angle of the target user's sight line.

[0149] In an implementation of this embodiment, the second determining unit 503 includes:

[0150] A second determining subunit is used to determine the face center point position information of the target user according to the face image of the target user;

[0151] The calculation subunit is used to calculate the terminal position information of the target robot according to the face center position information of the target user and the sight angle of the target user.

[0152] Furthermore, an embodiment of the present application also provides a robot interaction device, including: a processor, a memory, and a system bus;

[0153] The processor and the memory are connected via the system bus;

[0154] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementation methods of the above-mentioned robot interaction method.

[0155] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned robot interaction method.

[0156] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned robot interaction method.

[0157] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0158] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0159] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0160] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robot interaction method, characterized in that: include: Obtain the target user's facial image; Determine the sight angle of the target user by using the face image of the target user; Determining the terminal position information of the target robot according to the face image of the target user and the sight angle of the target user; Controlling the target robot to move to the end position; The step of using the face image of the target user to determine the sight angle of the target user includes: Inputting the face image of the target user into a pre-built sight line angle recognition model to identify the sight line angle of the target user; The sight angle recognition model includes a backbone network and a fully connected layer; the sight angle includes a vertical angle and a horizontal angle; The step of inputting the face image of the target user into a pre-built sight line angle recognition model to identify the sight line angle of the target user includes: Inputting the face image of the target user into the backbone network of the sight angle recognition model to obtain a backbone feature vector of the face image of the target user; Inputting the backbone feature vector of the target user's facial image into the fully connected layer of the sight line angle recognition model for angle regression processing to obtain the vertical angle and horizontal angle of the target user's sight line; The step of determining the terminal position information of the target robot according to the face image of the target user and the sight angle of the target user includes: Determine the face center position information of the target user according to the face image of the target user; The terminal position information of the target robot is calculated according to the face center position information of the target user and the sight angle of the target user.

2. The method according to claim 1, characterized in that Before acquiring the face image of the target user, the method further includes: Receive a preset start instruction, and control the target robot to enter an interactive mode according to the preset start instruction.

3. The method according to claim 2, characterized in that The preset start instruction includes a preset voice start instruction; the receiving the preset start instruction and controlling the target robot to enter the interactive mode according to the preset start instruction includes: Receive a preset voice start command, and determine the location information of a target user interacting with the target robot according to the microphone array to which the voice belongs; According to the position information of the target user, the target robot is controlled to rotate so that the face of the target user appears in the shooting device of the target robot, so as to interact with the target user.

4. The method according to claim 3, characterized in that The step of obtaining the face image of the target user includes: According to the position information of the target user and the current position information of the target robot, the target robot is controlled to rotate so that the distance between the target robot and the target user is within a preset range; The facial image of the target user is captured using the shooting device of the target robot.

5. The method according to claim 1, characterized in that The sight angle recognition model is constructed as follows: Obtaining facial images of sample users; Inputting the facial image of the sample user into an initial sight angle recognition model, and performing training using a preset objective function to generate the sight angle recognition model; Among them, the initial sight angle recognition model includes an initial backbone network and an initial fully connected layer.

6. The method according to claim 5, characterized in that The preset objective function is the mean square error function MSE.

7. A robot interaction device, characterized in that: include: A first acquisition unit, used to acquire a face image of a target user; A first determining unit, configured to determine a sight angle of the target user using a face image of the target user; A second determining unit is used to determine the terminal position information of the target robot according to the face image of the target user and the sight angle of the target user; A first control unit, used for controlling the target robot to move to the end position; The first determining unit is specifically configured to: Inputting the face image of the target user into a pre-built sight line angle recognition model to identify the sight line angle of the target user; The sight angle recognition model includes a backbone network and a fully connected layer; the sight angle includes a vertical angle and a horizontal angle; The first determining unit includes: A first input subunit, configured to input the face image of the target user into the backbone network of the sight angle recognition model to obtain a backbone feature vector of the face image of the target user; A second input subunit is used to input the backbone feature vector of the face image of the target user into the fully connected layer of the sight line angle recognition model for angle regression processing to obtain the vertical direction angle and horizontal direction angle of the sight line of the target user; The second determining unit includes: A second determining subunit is used to determine the face center point position information of the target user according to the face image of the target user; The calculation subunit is used to calculate the terminal position information of the target robot according to the face center position information of the target user and the sight angle of the target user.

8. A robot interaction device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 6.

Images