Robot control method and device, related equipment and computer program product

Abstract

The invention discloses a robot control method and device, related equipment and a computer program product. When an awakening instruction of a user is received, the pose of a robot is adjusted, so that a visual perception unit faces the position where the user is located, and visual information is collected through the visual perception unit; when a task voice instruction of a user is detected, voice instruction recognition is started, and before a user instruction intention is recognized, human body posture recognition is started so as to perform human body posture recognition on visual information collected by a visual perception unit, and when a posture interaction intention of the user is obtained, human body posture recognition is performed. The human body posture recognition result of each frame of image in the visual information within the time range can be backtracked according to the front and rear endpoint time of the voice segment corresponding to the task voice instruction, and the final human body posture recognition result is determined. According to the scheme, on one hand, the response speed can be increased, on the other hand, use habits of users are better met, and the success rate of man-machine interaction is increased.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and human-computer interaction, and more specifically, to a robot control method, apparatus, related equipment and computer program product. Background Art

[0002] With the development of technology, the use of interactive robots is becoming more and more widespread. Human-computer interaction can be separated from the remote control of remote platforms and allow robots to perform some autonomous behaviors.

[0003] During the human-computer interaction process, the user can give the machine a task voice command with certain body movements (such as gestures). For example, the user points to the location on the ground that needs to be cleaned and gives a voice command: "Clean the location I pointed to". Traditional solutions generally regard the human-computer interaction process as a serial process, that is, first recognize the user's voice command, and after obtaining the user's intention, further control the robot to adjust the posture so that the camera is aimed at the user, collect the user's image, and perform gesture recognition based on the collected image to determine the interaction information indicated by the user (such as the indicated location information). However, the user's usage habits are not necessarily serial execution. Some users usually speak while performing actions, or even perform actions first and then speak, and may stop performing actions after the voice command is finished. Since the execution of the algorithm takes time, it takes time from the end of the user's voice command to the machine recognizing the user's intention. When the traditional solution starts to start gesture recognition after obtaining the user's intention, the user may have put down the gesture, resulting in the inability to recognize the user's gesture, which in turn causes the interaction failure. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a robot control method, device, related equipment and computer program product to improve the success rate of human-computer interaction. The specific solution is as follows:

[0005] In a first aspect, a robot control method is provided, comprising:

[0006] In response to a user's wake-up command, the robot identifies the user's location and controls the robot to adjust its position so that a visual perception unit on the robot faces the user's location, and collects visual information through the visual perception unit;

[0007] When a user's task voice command is detected, voice command recognition is started, and before the user's command intention is recognized, human posture recognition is started to perform human posture recognition on each frame image in the visual information continuously collected by the visual perception unit to obtain a human posture recognition result for each frame image;

[0008] After obtaining the user instruction intention, if it is determined that the user instruction intention indicates that the user has a gesture interaction intention, then according to the front and rear endpoint times of the voice segment corresponding to the task voice instruction, the human body gesture recognition results of each frame image in the visual information within the front and rear endpoint time range are used to determine the final human body gesture recognition result;

[0009] The robot is controlled to perform an action matching the final human posture recognition result.

[0010] In one possible design, in another implementation of the first aspect of the embodiment of the present application, the time for starting human gesture recognition is synchronized with the time for starting voice command recognition.

[0011] In a possible design, in another implementation of the first aspect of the embodiment of the present application, starting human posture recognition to perform human posture recognition on each frame image in the visual information continuously collected by the visual perception unit to obtain a human posture recognition result for each frame image includes:

[0012] A sliding window selection is performed on each frame image in the visual information continuously collected by the visual perception unit, and human posture recognition is started for each continuous frame image in each sliding window to determine a human posture recognition result corresponding to each sliding window. The human posture recognition result includes whether each frame image in the sliding window has a set posture action, whether the posture action is stable, and interactive information indicated by the stable posture action.

[0013] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of determining the final human posture recognition result by using the human posture recognition result of each frame image in the visual information within the front and rear endpoint time range includes:

[0014] Determine each target sliding window within the time range of the front and rear endpoints, and a human posture recognition result corresponding to each target sliding window;

[0015] If there is at least one human posture recognition result corresponding to the target sliding window that contains interactive information indicated by a stable posture action, the final human posture recognition result is determined according to the interactive information indicated by the stable posture action contained in the human posture recognition results corresponding to each target sliding window.

[0016] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of determining the final human posture recognition result according to the interactive information indicated by the stable posture action contained in the human posture recognition result corresponding to each target sliding window includes:

[0017] The human posture recognition result of the last target sliding window in the time sequence among the target sliding windows whose human posture recognition results contain the interactive information indicated by the stable posture action is taken as the final human posture recognition result.

[0018] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the interaction information indicated by the stable posture action is the coordinates of the pointing point; then, according to the interaction information indicated by the stable posture action contained in the human posture recognition result corresponding to each target sliding window, the process of determining the final human posture recognition result includes:

[0019] According to the pointing point coordinates indicated by the stable gesture actions contained in the human gesture recognition results corresponding to each target sliding window, the mean or median of the pointing point coordinates is calculated as the final human gesture recognition result.

[0020] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the method further includes:

[0021] If the human body posture recognition results corresponding to each of the target sliding windows indicate that there is no set posture action, outputting first prompt information, wherein the first prompt information is used to prompt the user to perform the set posture action;

[0022] If the human posture recognition results corresponding to the target sliding windows do not contain interactive information indicated by a stable posture action, but the human posture recognition result of the target sliding window indicates that the posture action is unstable, then the second prompt information is output, and the second prompt information is used to prompt the user to keep a stable posture.

[0023] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of starting human posture recognition for each consecutive frame image in each sliding window and determining the human posture recognition result corresponding to each sliding window includes:

[0024] Initiate human posture recognition for each frame of image in each sliding window to determine whether the image contains a set posture action;

[0025] If it is determined that the image has a set posture action, further determining whether the set posture action is stable according to each frame image in the sliding window;

[0026] If it is determined that the set posture action is stable, then the interaction information indicated by the stable posture action is further determined according to each frame image.

[0027] In a possible design, in another implementation of the first aspect of the embodiment of the present application, in response to a wake-up instruction of the user, a process of identifying the location of the user includes:

[0028] The sound source is localized according to the voice segment corresponding to the wake-up command to obtain the direction of the sound source;

[0029] The visual information of the sound source orientation is collected by the visual perception unit on the robot, and whether the user is recognized is determined based on the visual information. If not, the robot is controlled to rotate along the first rotation direction until the user's position is recognized or one rotation is completed.

[0030] In one possible design, in another implementation of the first aspect of the embodiments of the present application, the human posture recognition includes hand gesture recognition.

[0031] In a second aspect, a robot control device is provided, comprising:

[0032] A wake-up instruction response unit, used to respond to the user's wake-up instruction, identify the user's location and control the robot to adjust the posture so that the visual perception unit on the robot faces the user's location, and collect visual information through the visual perception unit;

[0033] A human posture recognition unit, used to start voice command recognition when a user's task voice command is detected, and start human posture recognition before recognizing the user's command intention, so as to perform human posture recognition on each frame image in the visual information continuously collected by the visual perception unit, and obtain a human posture recognition result for each frame image;

[0034] A recognition result backtracking unit is used to determine the final human gesture recognition result based on the human gesture recognition results of each frame image in the visual information within the time range of the front and back endpoints of the voice segment corresponding to the task voice command, if it is determined that the user's instruction intention indicates that the user has an intention of gesture interaction after obtaining the user's instruction intention;

[0035] The action execution unit is used to control the robot to execute an action matching the final human posture recognition result.

[0036] A third aspect provides a robot, comprising:

[0037] Equipment body;

[0038] A speech unit, a visual perception unit, a travel unit and a control unit are arranged on the device body;

[0039] The speech unit is used to receive speech data;

[0040] The visual perception unit is used to collect visual information;

[0041] The traveling unit is used to adjust the posture of the robot under the control of the control unit;

[0042] The control unit is used to execute each step of the robot control method described in any one of the first aspects of the present application.

[0043] In a fourth aspect, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the robot control method described in any one of the first aspects of the present application are implemented.

[0044] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the various steps of the robot control method described in any one of the aforementioned first aspects of the present application.

[0045] It can be seen from the above technical solution that the present application does not start the recognition and processing of human posture after recognizing the user's task voice command and obtaining the user's command intention, but adjusts the robot's posture so that the visual perception unit faces the user's position when receiving the user's wake-up command, and starts to collect visual information through the visual perception unit. On this basis, when the user's task voice command is detected, the voice command recognition is started, and before the user's command intention is recognized, the human posture recognition is started to perform human posture recognition on each frame image in the visual information continuously collected by the visual perception unit to obtain the recognition result of each frame image. After obtaining the user's command intention, if it is determined that the intention indicates that the user has a gesture interaction intention, the human posture recognition results of each frame image in the visual information within the time range can be traced back according to the front and rear endpoint times of the voice segment corresponding to the task voice command, and the final human posture recognition result can be determined. Obviously, compared with the method of starting to collect visual information and perform human posture recognition after obtaining the user's command intention, the present application solution can recognize human posture in advance, which can speed up the response speed on the one hand, and on the other hand, it is more in line with the user's usage habits and improves the success rate of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0047] Figure 1 A schematic diagram of an implementation system architecture of a robot control method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of a robot control method flow chart provided in an embodiment of the present application;

[0049] Figure 3A schematic flow chart of a method for adjusting the posture of a robot in response to a wake-up instruction provided in an embodiment of the present application;

[0050] Figure 4 A schematic flow chart of another method for adjusting the posture of a robot in response to a wake-up instruction provided in an embodiment of the present application;

[0051] Figure 5 A schematic flow chart of another method for adjusting the posture of a robot in response to a wake-up instruction provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the correspondence between a user task voice instruction and an image human posture recognition result provided in an embodiment of the present application;

[0053] Figure 7 This is a schematic diagram of the structure of a robot control device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] The essence of robot interaction solutions is to make people's lives more convenient, and more accurate human-machine interaction solutions can bring better experience. Existing solutions focus more on improving the recognition accuracy of related algorithms, such as improving the accuracy of voice recognition, intention recognition, and improving the accuracy of image-based human pointing point position recognition.

[0056] The traditional human-computer interaction process is generally to rotate towards the user based on the sound source localization algorithm after receiving the user's gesture interaction voice command (for example: the user issues the command "clean the place I point to"), then collect the user's image, and perform gesture recognition (such as gesture recognition) to determine the location pointed by the user. This serial interaction response process, on the one hand, increases the extra waiting time for the user after issuing the voice command. On the other hand, it does not conform to the actual usage habits of users. The user's interaction process is not necessarily serial. Some users usually point while speaking or point before speaking. Even because the user assumes that the robot is "listening and watching", they will put down the gesture immediately or just a moment after the voice command is issued. Since the voice recognition process takes time, it takes a certain amount of time from the end of the user's voice command to the machine recognizing the user's command intention. When the gesture recognition is started after the user's command intention is obtained, the user may have put down the gesture. Missing the user's gesture action leads to the inability to detect the user's interaction information, such as the position coordinates indicated by the gesture, which leads to the failure of human-computer interaction.

[0057] To this end, the present application provides a robot control solution. Unlike existing solutions, the present application aims to solve the defects of existing solutions, improve the success rate of human-computer interaction, and provide users with a more convenient and comfortable interaction experience at least through improvements in the human-computer interaction process.

[0058] The robot control solution provided in this application can be applied to control various types of robots. The robot referred to in this application refers to an electronic device that can move autonomously and can achieve intelligent control. In some scenarios, the robot can be a robot with sweeping and cleaning functions. Figure 1 The robot sweeping robot is used as an example to illustrate the structure of the robot control system of the present application, but it does not mean that the robot provided by the present application can only be implemented as a sweeping robot. In some other scenarios, the robot of the present application can also be a warehouse logistics robot, such as a freight robot, a goods delivery robot, or a robot waiter, such as a hotel welcoming robot, a food serving robot, a shopping guide robot in a shopping mall, etc.

[0059] The robots listed above are only used for exemplary description, and this embodiment includes but is not limited to them.

[0060] The robot control method provided in this application can be applied to Figure 1 As shown in the system architecture, the system may include a robot 100 and a control terminal 200 .

[0061] The robot 100 or the control terminal 200 can be used alone to execute the robot control method provided in the embodiment of the present application. In addition, the robot 100 and the control terminal 200 can also be used in collaboration to execute the robot control method provided in the embodiment of the present application.

[0062] In a possible embodiment, the robot 100 may send data to the control terminal 200, and receive the processing result sent by the control terminal 200. Exemplarily, the robot 100 sends the received user's voice command to the control terminal 200, and the control terminal 200 recognizes the voice command and returns the recognized user command intention to the robot 100. The robot 100 sends the visual information collected by the visual perception unit to the control terminal 200, and the control terminal 200 starts the human posture recognition, obtains the human posture recognition result of each frame image, and returns it to the robot 100. The robot 100 determines the final human posture recognition result based on the returned human posture recognition result, and performs the matching action.

[0063] In another possible embodiment, the robot 100 can independently complete the control process, that is, the robot 100 can be locally configured with a voice recognition engine and a human posture recognition engine, and then can locally complete the recognition of voice commands and human postures, and complete the control process of the robot based on the recognition results.

[0064] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, usage scenarios, etc. of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0065] The present application provides a robot control method, which is applied to Figure 1 Taking the computer device in the example as an example, the computer device can be Figure 1 The robot 100 in Figure 2 , the robot control method specifically includes the following steps:

[0066] Step S100: In response to a wake-up instruction from the user, the visual perception unit of the robot is directed toward the user and collects visual information.

[0067] Specifically, the user may initiate a wake-up instruction in the form of voice, and the wake-up instruction may be a voice instruction including a set wake-up word.

[0068] After the user initiates a wake-up command, the user's location can be identified through sound source positioning and / or vision-based human recognition, and then the robot is controlled to adjust its posture so that the visual perception unit on the robot is facing the user's location, and the visual perception unit is started to start collecting visual information.

[0069] The visual perception unit may be composed of at least one camera module or at least one area array ranging module. The camera module may use an RGB camera or other types of camera modules. In a possible implementation, in order to ensure both recognition and ranging accuracy, the present application may use at least two camera modules, or at least one camera module in combination with an area array ranging module.

[0070] The robot of the present application may be provided with a voice unit, which may be composed of at least one microphone. In some possible implementations, the voice unit may be composed of at least three microphones, so as to estimate the relative direction of the user and the robot through the time difference from the user's voice command to the three microphones, accelerate the robot to face the user's position, or accurately face the speaker who issued the voice command when there are multiple people in the space.

[0071] Compared with the prior art, in which the robot posture is adjusted, the camera is started for image acquisition and human posture recognition only after the user issues a specific task voice command and obtains the user's command intention through voice recognition, this application pre-stages the processing process of adjusting the robot posture to align with the user and starting the visual perception unit to collect visual information, that is, the robot posture adjustment and the visual perception unit collect visual information processing steps are executed after the user issues a wake-up command. It can speed up the response speed and prepare for starting human posture recognition in the subsequent steps.

[0072] Step S110: when the user's task voice command is detected, voice command recognition is started, and before the user's command intention is recognized, human posture recognition is started to obtain the human posture recognition result corresponding to the visual information.

[0073] Specifically, after waking up the robot, the user can further issue task voice instructions. The task voice instructions can be of multiple types, such as task instructions that characterize the user's intention of gesture interaction, such as gesture interaction voice instructions. In addition, they can also be task instructions with other non-gesture interaction intentions, such as voice instructions to order the sweeping robot to return to the base station.

[0074] Considering that the user's task voice command may be a task command with gesture interaction intention, the user will issue the task voice command in conjunction with body movements. In order to avoid the existing solution that performs intent recognition on the task voice command and starts the human body gesture recognition process only after obtaining the user's task intention, the user may have stopped the body movement at this time, resulting in the inability to recognize the user's interaction intention. In this step, when the user's task voice command is detected, the voice command recognition algorithm is first started, that is, the task voice command is voice recognized and the intention is recognized. Secondly, before the voice command recognition algorithm processing process obtains the user's command intention, the human body gesture recognition process is further started to perform human body gesture recognition on each frame of the visual information continuously collected by the visual perception unit to obtain the human body gesture recognition result for each frame of the image.

[0075] In a possible implementation, the time of starting human gesture recognition can be synchronized with the time of starting voice command recognition, that is, when the user's task voice command is detected, the voice command recognition and human gesture recognition processes are started synchronously. The "synchronization" here can be understood as the same time, and the system response time error can also be considered to ensure that the start time of the two recognition processes is within a very short time.

[0076] This step changes the existing method of using voice command recognition and human posture recognition as a serial processing flow, and advances the human posture recognition process (compared to the existing method of starting the human posture recognition process only after voice command recognition obtains the user's command intention), which is more in line with the user's usage habits, that is, issuing voice commands while performing actions, thereby improving the interaction success rate.

[0077] In this step, human posture recognition is started to perform human posture recognition on each frame image in the visual information continuously collected by the visual perception unit to obtain a human posture recognition result for each frame image.

[0078] Among them, the gestures to be recognized by human gesture recognition can be determined according to the interactive actions specified by the business logic. For example, if the pre-agreed human-computer interactive actions are gesture actions, then the gestures to be recognized by human gesture recognition here are the pre-set gesture actions of various types. In addition, human-computer interactive actions can also include actions of other parts of the body, such as leg movements, toe movements, head movements, etc. In the subsequent embodiments of this application, only gesture actions are used as examples for explanation.

[0079] Step S120: When it is determined that the user has a gesture interaction intention, based on the front and back endpoint times of the voice segment corresponding to the task voice instruction, the human gesture recognition results of each frame image within the time range are traced back to determine the final human gesture recognition result.

[0080] Specifically, voice command recognition is started in the previous step to identify the user's command intention corresponding to the user's task voice command. After identifying the user's command intention, if it is further determined that the user's command intention represents the user's intention of gesture interaction, then the front and back endpoint times of the voice segment corresponding to the task voice command can be determined at this time, and then the human posture recognition results of each frame image within the front and back endpoint time range are used to determine the final human posture recognition result.

[0081] The result of voice command recognition may include the front-end point, back-end point, text information corresponding to the voice, and the user's command intention. When the front-end point of the voice is recognized, the time of the front-end point can be recorded and the voice segment can be retained; when the back-end point of the voice is recognized, the time of the back-end point can be recorded and the voice segment capture can be ended. The captured voice segment is sent to the voice recognition model to obtain the recognition text corresponding to the voice task command, and the recognition text is further sent to the intention recognition model to obtain the user's command intention.

[0082] Among them, the speech recognition model and the intent recognition model can adopt the currently existing models, or can adopt the models with speech recognition and intent recognition functions developed in the future, which is not limited in this case.

[0083] Step S130: Control the robot to perform an action matching the final human posture recognition result.

[0084] Specifically, after the final human posture recognition result is determined in the above steps, the interaction information indicated by the user through the gesture action can be obtained. Taking the gesture action as an example, the position coordinates indicated by the user's gesture can be obtained. On this basis, the robot can be controlled to perform an action that matches the final human posture recognition result.

[0085] For example, a user interacts with a sweeping robot through gestures. The user points to the ground through gestures and issues a task voice command "clean the area I point to". After the final human gesture recognition result is obtained using the method of this embodiment, the coordinates of the ground position pointed to by the user's gesture can be determined, and then the robot can be controlled to move to the coordinates of the ground position and perform the cleaning task.

[0086] The robot control method provided in the embodiment of the present application does not start the recognition and processing of human posture after recognizing the user's task voice command and obtaining the user's command intention, but adjusts the robot's posture so that the visual perception unit faces the user's position when receiving the user's wake-up command, and starts to collect visual information through the visual perception unit. On this basis, when the user's task voice command is detected, the voice command recognition is started, and before the user's command intention is recognized, the human posture recognition is started to perform human posture recognition on each frame image in the visual information continuously collected by the visual perception unit to obtain the recognition result of each frame image. After obtaining the user's command intention, if it is determined that the intention indicates that the user has a gesture interaction intention, the human posture recognition results of each frame image in the visual information within the time range can be traced back according to the front and rear endpoint times of the voice segment corresponding to the task voice command, and the final human posture recognition result can be determined. Obviously, compared with the method of starting to collect visual information and perform human posture recognition after obtaining the user's command intention, the present application solution can recognize human posture in advance, which can speed up the response speed on the one hand, and is more in line with the user's usage habits on the other hand, thereby improving the success rate of human-computer interaction.

[0087] In some embodiments of the present application, the process of identifying the user's location in response to the user's wake-up command in the aforementioned step S100 is described.

[0088] In this embodiment, a sound source localization algorithm may be used to identify the user's location, or a vision-based human figure recognition algorithm may be used to identify the user's location. Alternatively, the sound source localization algorithm and the human figure recognition algorithm may be combined to identify the user's location.

[0089] A possible implementation method is to combine Figure 3 As shown, the following steps may be included:

[0090] Step S11: localize the sound source according to the voice segment corresponding to the wake-up instruction to obtain the direction of the sound source.

[0091] Step S12: collect visual information of the sound source orientation through the visual perception unit on the robot, and determine whether the user is recognized based on the visual information.

[0092] Here, a human figure recognition algorithm can be used to detect whether a user is recognized based on visual information. The human figure recognition algorithm can use a pre-trained deep learning model. The visual information can be a plane image (such as an RGB image, an infrared image, a grayscale image, etc.) or a point cloud image obtained by a surface array ranging module.

[0093] If the user is not recognized based on the visual information, step S13 may be executed; if the user is recognized based on the visual information, step S14 may be executed.

[0094] Step S13: Control the robot to rotate in situ along the first rotation direction. During the rotation process, the robot continues to detect whether the user is recognized based on the collected visual information until the user's position is recognized, or the robot returns to the original position after rotating one circle.

[0095] Step S14: Get the user's location.

[0096] Another possible implementation is to combine Figure 4 As shown, the following steps may be included:

[0097] Step S21: localize the sound source according to the voice segment corresponding to the wake-up instruction to obtain a first angle of the sound source relative to the robot.

[0098] Specifically, the process of sound source localization is to estimate the relative direction of the sound source to the robot through the time difference between the voice segments corresponding to the wake-up command reaching different microphones in the microphone array, and then obtain the first angle of the relative direction relative to the robot's orientation.

[0099] Step S22: determine whether the first threshold range of the first angle is within the field of view of the robot visual perception unit to decide whether the robot posture orientation needs to be rotated.

[0100] In a possible implementation, according to the calibration information of the installation position of the robot visual perception unit on the robot, the angle of the human figure in the image captured by the visual perception unit can be converted to the robot coordinate system, that is, the relative direction of the human figure in the image to the robot can be obtained. Optionally, the X-axis direction of the robot coordinate system can be directed toward the positive forward direction of the robot travel unit. The visual perception unit is generally installed on the front side of the robot body, facing the X-axis direction of the robot coordinate system. The angle of the human figure in the image is equal to the angle of the X-axis direction of the robot coordinate system.

[0101] The first threshold range can be preset, for example, the value is [-22.5°, 22.5°]. It can be understood that the smaller the first threshold range is, the higher the requirements for the sound source localization algorithm is, but it can better avoid the robot heading towards the wrong target humanoid when there are multiple people in the space.

[0102] In this step, if the first threshold range of the first angle is determined to be within the field of view of the robot's visual perception unit, it can be determined that the robot posture does not need to be rotated. Otherwise, in order to ensure that the visual perception unit can capture images in the first angle direction, the robot posture needs to be rotated.

[0103] When the judgment result is that rotation is required, step S23 can be executed, otherwise, step S24 is executed.

[0104] Step S23, determining a first rotation direction, and rotating the robot along the first rotation direction until the visual perception unit faces a first angle.

[0105] The first rotation direction may be clockwise or counterclockwise.

[0106] In a possible implementation, the first rotation direction may be a direction that causes the robot to rotate to a direction with a shorter stroke facing the first angle.

[0107] Step S24, determine whether there is a human figure within the first threshold range of the first angle, if not, execute step S25, if yes, execute step S26.

[0108] It is understandable that, in actual application scenarios, there may be no humanoid user within the first threshold range of the first angle where the sound source is located, obtained by the sound source localization algorithm. For example, when the background noise or the robot's self-noise is large, the sound source localization algorithm may be interfered by the noise, resulting in a certain deviation in the result, and there is no user within the first threshold range of the first angle. At this time, in an optional processing method, step S25 can be executed. Of course, it is also possible to remain motionless and wait for subsequent instructions from the user. If it is determined that there is a human figure within the first threshold range of the first angle, it means that the image of the user can be collected at present. At this time, the user's location can be obtained, and the current posture of the robot can be kept unchanged, or step S26 can be executed.

[0109] Step S25, continue to rotate along the first rotation direction to determine whether there is a human figure in the field of view of the robot's visual perception unit until the robot faces the first angle again.

[0110] If a human figure is found during the rotation process, the user's location can be obtained, and the current position of the robot can be kept unchanged, or step S26 can be executed. If no human figure is found during the rotation process, step S27 can be executed.

[0111] Step S26, rotating the robot until the visual perception unit faces the direction of the human figure.

[0112] Specifically, in order to improve the quality of the image captured by the visual perception unit, the robot can be rotated until the visual perception unit faces the direction of the human figure.

[0113] Step S27, waiting for subsequent instructions from the user.

[0114] Specifically, if the user's location is not detected after the above steps, the user's subsequent instructions can be continued to be waited for.

[0115] Another possible implementation is that when the robot is working in a specific mode, it can be assumed that the sound source positioning angle is inaccurate. For example, when the robot is in the maximum suction mode, due to the large self-noise of the robot, it can be assumed that the result obtained by using the sound source positioning at this time is inaccurate. Alternatively, when the robot is not equipped with more than three microphones, the sound source positioning algorithm cannot be used. In this embodiment, only the human figure recognition algorithm based on vision can be used to identify the user's location, combined with Figure 5 As shown, the following steps may be specifically included:

[0116] Step S31, responding to the wake-up instruction and starting human figure recognition.

[0117] Step S32: determine whether there is a human figure in the field of view of the robot's visual perception unit. If so, the user's location can be obtained, and the robot's current position can be kept unchanged, or step S33 can be executed. If no human figure is found, step S34 can be executed.

[0118] Step S33, rotating the robot until the visual perception unit faces the direction of the human figure.

[0119] Specifically, in order to improve the quality of the image captured by the visual perception unit, the robot can be rotated until the visual perception unit faces the direction of the human figure.

[0120] Step S34, continue to rotate along the second rotation direction to determine whether there is a human figure in the field of view of the robot's visual perception unit until the robot returns to the starting position again.

[0121] If a human figure is found during the rotation, the user's location can be obtained, and the current position of the robot can be kept unchanged, or step S33 can be executed. If no human figure is found during the robot's rotation of 360 degrees and returning to the starting position, step S35 can be executed.

[0122] The second rotation direction may be clockwise or counterclockwise.

[0123] Step S35: Wait for subsequent instructions from the user.

[0124] Specifically, if the user's location is not detected after the above steps, the user's subsequent instructions can be continued to be waited for.

[0125] The above embodiments introduce several implementation methods of identifying the user's location in response to a user wake-up command, and controlling the robot to adjust the posture so that the visual perception unit is facing the user's location. Any one of the methods can be used in actual applications, or other variations can be used, which will not be described one by one in this application.

[0126] In some embodiments of the present application, the process of starting human posture recognition in the aforementioned step S110 and obtaining the human posture recognition result corresponding to the visual information is described.

[0127] In a possible implementation, human posture recognition may be performed on each image frame in the visual information continuously collected by the visual perception unit to obtain a human posture recognition result for each frame of the image.

[0128] Combination Figure 6 As shown, in another possible implementation, this embodiment can perform sliding window selection on each frame image in the visual information continuously collected by the visual perception unit, and start human posture recognition on each continuous frame image in each sliding window to determine the human posture recognition result corresponding to each sliding window.

[0129] When sliding window selection is performed on each frame image in the visual information, selection can be performed according to the set window length and sliding step length. The window length is greater than or equal to 2 frames of images ( Figure 6 In the specific implementation, taking the visual perception unit of 25fps as an example, the window length of the sliding window can be set to 20 frames. The sliding step length can be equal to the window length, that is, the sliding windows can not overlap.

[0130] When human posture recognition is performed on each frame image in each sliding window, it can be determined whether the image in the sliding window has a set posture action (referred to as whether there is action); if the image has a set posture action, it can be further determined whether the posture action is stable based on each frame image in the sliding window.

[0131] Specifically, when judging whether the posture and action of each frame image in the sliding window is stable, the bone node detection technology can be used to extract the coordinates of multiple joints on the limbs in the image space, and compare whether the position change of the same joint point in the image space exceeds the set distance threshold between consecutive frames of images. If it exceeds, the posture and action is considered unstable, and if it does not exceed, the posture and action is considered stable. In addition, the image time series modeling method can also be used to predict whether the posture and action of each frame image in the sliding window is stable through a deep learning algorithm.

[0132] Furthermore, when it is determined that the posture and action of each frame image in the sliding window is stable, the interactive information indicated by the stable posture and action can also be determined according to each frame image in the difference.

[0133] In different interactive tasks, the interactive information indicated by gestures may be different. For example, gestures can indicate location information, such as indicating the ground area that the sweeping robot needs to clean; indicate operation instructions, such as instructing the robot to grab, put down, stop, start, confirm and cancel operation instructions; indicate state control information, such as instructing the robot to change the operating speed, adjust the working mode, etc.

[0134] In this embodiment, the process of determining the position information indicated by the stable posture action is described by taking the interaction information as the indicated position information as an example.

[0135] In an optional implementation, multiple joint points can be extracted from each frame image and mapped to the physical space based on each frame image in the sliding window. For example, the positions of eyes and fingers are extracted and mapped to the physical space, a straight line passing through the eyes and fingers is calculated, and the intersection of the straight line and the ground is calculated to obtain the coordinates of the pointing point. The coordinates of the pointing point are calculated for each frame image in the sliding window, and the average or median of the coordinates of each pointing point is taken as the coordinates of the pointing point corresponding to the sliding window.

[0136] In another optional implementation, a deep learning algorithm can be used to pre-train a pointing point prediction model, and then each frame image in the sliding window is input into the model to obtain the pointing point coordinates corresponding to the image predicted by the model. Further, the average or median of the pointing point coordinates corresponding to each frame image is taken as the pointing point coordinates corresponding to the sliding window.

[0137] The method introduced in this embodiment performs sliding window selection on the visual information collected by the visual perception unit, divides each continuous frame image into several sliding windows, and then identifies the human posture recognition result corresponding to each sliding window according to the sliding windows, thereby reducing the error of the single-frame image recognition result.

[0138] On this basis, the process of step S120, which is to trace back the human posture recognition results of each frame image within the time range according to the front and rear endpoint times of the voice segment corresponding to the task voice instruction, and determine the final human posture recognition result, is introduced.

[0139] Combination Figure 6 As shown, according to the front and rear endpoints of the voice segment corresponding to the task voice command, each target sliding window within the time range of the front and rear endpoints can be located, as well as the human posture recognition result corresponding to each target sliding window.

[0140] If there is at least one human posture recognition result corresponding to a target sliding window that contains interactive information indicated by a stable posture action, the final human posture recognition result can be determined according to the interactive information indicated by the stable posture action contained in the human posture recognition results corresponding to each target sliding window.

[0141] Among them, if the time of the start frame of a sliding window is before the front end point, and the time of the last frame is after the front end point, then the sliding window can be processed as a target sliding window, or it can be processed as a target sliding window. Similarly, if the time of the start frame of a sliding window is before the rear end point, and the time of the last frame is after the rear end point, then the sliding window can be processed as a target sliding window, or it can be processed as a target sliding window. In the following embodiments of the present application, the above two sliding windows can be processed as target sliding windows.

[0142] In a possible implementation, the process of determining the final human posture recognition result according to the interactive information indicated by the stable posture action contained in the human posture recognition result corresponding to each target sliding window can be to take the human posture recognition result of the last target sliding window in time sequence among the target sliding windows whose human posture recognition results contain the interactive information indicated by the stable posture action as the final human posture recognition result.

[0143] In addition, the interactive information indicated by the stable posture movements in each target sliding window can be comprehensively considered, and the final human posture recognition result can be calculated according to the set fusion strategy, that is, the final interactive information can be calculated.

[0144] Take the interactive information as the coordinates of the pointing point as an example to illustrate:

[0145] In the present application, the mean or median of the pointing point coordinates indicated by the stable posture movements contained in the human posture recognition results corresponding to each target sliding window can be calculated as the final human posture recognition result.

[0146] Combination Figure 6 As shown, there are multiple target sliding windows within the time range of the front end point and the back end point, and the sliding windows are numbered from left to right, namely sliding window 1, sliding window 2, sliding window 3, sliding window 4, sliding window 5, sliding window 6, and sliding window 7. Among them, sliding windows 2-6 are target sliding windows.

[0147] The user corresponding to sliding window 1 has not started to take action yet.

[0148] Slide window 2 and slide window 3 belong to the process of user raising hand, there is gesture action, but the action is unstable.

[0149] There is a gesture action on the sliding window 4, and the action is stable, and the pointing point P0 corresponding to the gesture can be calculated.

[0150] There is a gesture action on the sliding window 5, and the action is stable, and the pointing point P1 corresponding to the gesture can be calculated.

[0151] Sliding window 6 belongs to the process of the user lowering his arm. There is a gesture action, but the action is unstable.

[0152] Slide window 7 indicates that the user has put down the gesture and the gesture action cannot be detected. In addition, according to the end point time of the voice signal, when the user stops making actions a moment after speaking the voice command, if the traditional solution is used, the user has put down the gesture by the time the voice recognition determines the user's command intention, which may result in the inability to detect the gesture.

[0153] According to the solution of the present application, it can be determined that there are two target sliding windows containing interactive information (i.e., pointing points) indicated by stable actions, and the final human posture recognition result can be determined based on the interactive information indicated by the stable actions contained in the two target sliding windows, that is: the final pointing point is determined based on the pointing point P0 corresponding to sliding window 4 and the pointing point P1 corresponding to sliding window 5.

[0154] For example, the pointing point P1 corresponding to the sliding window 5 which is later in time sequence is taken as the final pointing point. Alternatively, the mean or median of the pointing point P0 and the pointing point P1 can be taken as the final pointing point.

[0155] It should be noted that in the above steps, based on the front and rear endpoints of the voice segment corresponding to the task voice command, each target sliding window within the time range of the front and rear endpoints is located, and on the basis of the human posture recognition result corresponding to each target sliding window:

[0156] If it is determined that the human posture recognition results corresponding to each target sliding window all indicate that there is no set posture action, it can be considered that the user has not made the set posture action, and the first prompt information can be output at this time to prompt the user to make the set posture action. Taking the finger-sweeping control process of the sweeping robot as an example, the first prompt information can be "Please point to the location that needs to be cleaned with your hand."

[0157] In another case, if it is determined that the human posture recognition results corresponding to each target sliding window do not contain the interaction information indicated by the stable posture action, but the human posture recognition results of the target sliding window indicate that the posture action is unstable, it can be considered that the user has made the set posture action, but the action process is unstable, and it is impossible to calculate accurate interaction information based on the unstable posture action. Therefore, at this time, the second prompt information can be output to prompt the user to keep the posture stable. For example, for the process of indicating the cleaning position through gestures, if it is determined that the user's gesture action is unstable, the second prompt information can be output, such as "Please keep the current gesture stable."

[0158] The robot control method provided by the above embodiments of the present application advances the recognition process of human posture movements compared to the traditional solution, such as performing the recognition of human posture movements while recognizing the user's task voice command. In this way, after obtaining the user's intention through voice recognition, if it is determined that the intention indicates the existence of gesture interaction intention, the human posture recognition results of each frame image within the time range can be traced back according to the front and back endpoint times of the voice segment corresponding to the task voice command, and the final human posture recognition result can be determined. The use of the present application solution can improve the response speed of human-computer interaction and increase the success rate of human-computer interaction.

[0159] The robot control device provided in the embodiment of the present application is described below. The robot control device described below and the robot control method described above can be referenced to each other.

[0160] See also Figure 7 , Figure 7 This is a schematic diagram of the structure of a robot control device disclosed in an embodiment of the present application.

[0161] like Figure 7 As shown, the device may include:

[0162] A wake-up instruction response unit 11 is used to respond to the user's wake-up instruction, identify the user's location and control the robot to adjust the posture so that the visual perception unit on the robot faces the user's location, and collect visual information through the visual perception unit;

[0163] A human posture recognition unit 12 is used to start voice command recognition when a user's task voice command is detected, and start human posture recognition before recognizing the user's command intention, so as to perform human posture recognition on each frame image in the visual information continuously collected by the visual perception unit, and obtain a human posture recognition result for each frame image;

[0164] The recognition result backtracking unit 13 is used to determine the final human gesture recognition result based on the front and back endpoint times of the voice segment corresponding to the task voice command and the human gesture recognition results of each frame image in the visual information within the front and back endpoint time range after obtaining the user instruction intention, if it is determined that the user instruction intention indicates that the user has a gesture interaction intention;

[0165] The action execution unit 14 is used to control the robot to execute an action matching the final human posture recognition result.

[0166] In a possible implementation, the time of starting the human posture recognition is synchronized with the time of starting the voice command recognition.

[0167] In a possible implementation, the human posture recognition unit starts human posture recognition to perform human posture recognition on each frame of the visual information continuously collected by the visual perception unit to obtain a human posture recognition result for each frame of the image, including:

[0168] A sliding window selection is performed on each frame image in the visual information continuously collected by the visual perception unit, and human posture recognition is started for each continuous frame image in each sliding window to determine a human posture recognition result corresponding to each sliding window. The human posture recognition result includes whether each frame image in the sliding window has a set posture action, whether the posture action is stable, and interactive information indicated by the stable posture action.

[0169] In a possible implementation, the recognition result backtracking unit uses the human posture recognition results of each frame image in the visual information within the front and rear endpoint time range to determine the final human posture recognition result, including:

[0170] Determine each target sliding window within the time range of the front and rear endpoints, and a human posture recognition result corresponding to each target sliding window;

[0171] If there is at least one human posture recognition result corresponding to the target sliding window that contains interactive information indicated by a stable posture action, the final human posture recognition result is determined according to the interactive information indicated by the stable posture action contained in the human posture recognition results corresponding to each target sliding window.

[0172] In a possible implementation, the recognition result backtracking unit determines the final human posture recognition result according to the interactive information indicated by the stable posture action contained in the human posture recognition result corresponding to each target sliding window, including:

[0173] The human posture recognition result of the last target sliding window in the time sequence among the target sliding windows whose human posture recognition results contain the interactive information indicated by the stable posture action is taken as the final human posture recognition result.

[0174] In another possible implementation, the interactive information indicated by the stable gesture action is the coordinates of the pointing point; then the recognition result backtracking unit determines the final human gesture recognition result according to the interactive information indicated by the stable gesture action contained in the human gesture recognition result corresponding to each target sliding window, including:

[0175] According to the pointing point coordinates indicated by the stable gesture actions contained in the human gesture recognition results corresponding to each target sliding window, the mean or median of the pointing point coordinates is calculated as the final human gesture recognition result.

[0176] In a possible implementation, the device of the present application may further include:

[0177] a prompt output unit, configured to output first prompt information for prompting a user to perform a set gesture action if it is determined that the human gesture recognition results corresponding to each of the target sliding windows indicate that no set gesture action exists, based on the final human gesture recognition result obtained by the recognition result backtracking unit;

[0178] If it is determined that none of the human posture recognition results corresponding to the target sliding windows contain interactive information indicated by a stable posture action, but the human posture recognition result of the target sliding window indicates that the posture action is unstable, then second prompt information is output, and the second prompt information is used to prompt the user to keep a stable posture.

[0179] In a possible implementation, the human posture recognition unit starts human posture recognition for each consecutive frame image in each sliding window, and determines the human posture recognition result corresponding to each sliding window, including:

[0180] Initiate human posture recognition for each frame of image in each sliding window to determine whether the image contains a set posture action;

[0181] If it is determined that the image has a set posture action, further determining whether the set posture action is stable according to each frame image in the sliding window;

[0182] If it is determined that the set posture action is stable, then the interaction information indicated by the stable posture action is further determined according to each frame image.

[0183] In a possible implementation, the process of the wake-up instruction response unit identifying the location of the user in response to the wake-up instruction of the user includes:

[0184] The sound source is localized according to the voice segment corresponding to the wake-up command to obtain the direction of the sound source;

[0185] The visual information of the sound source orientation is collected by the visual perception unit on the robot, and whether the user is recognized is determined based on the visual information. If not, the robot is controlled to rotate along the first rotation direction until the user's position is recognized or one rotation is completed.

[0186] In a possible implementation, the human posture recognition includes hand gesture recognition.

[0187] In some embodiments of the present application, a robot is further provided, comprising:

[0188] Equipment body;

[0189] A speech unit, a visual perception unit, a travel unit and a control unit are arranged on the device body;

[0190] The voice unit is used to receive voice data;

[0191] The visual perception unit is used to collect visual information;

[0192] The traveling unit is used to adjust the posture of the robot under the control of the control unit;

[0193] The control unit is used to implement each step of the robot control method of any of the aforementioned embodiments.

[0194] The visual perception unit may be composed of at least one camera module or at least one area array ranging module. The camera module may use an RGB camera or other types of camera modules. In a possible implementation, in order to ensure both recognition and ranging accuracy, the present application may use at least two camera modules, or at least one camera module in combination with an area array ranging module.

[0195] The speech unit may be composed of at least one microphone. In some possible implementations, the speech unit may be composed of at least three microphones, so as to estimate the relative direction of the user and the robot through the time difference from the user's voice command to the three microphones, speed up the robot to face the user's position, or accurately face the speaker who issued the voice command when there are multiple people in the space.

[0196] The robot provided by the present application, wherein the control unit can execute the robot control method introduced in the aforementioned embodiment, can speed up the response speed, and on the other hand, is more in line with the user's usage habits, thereby improving the success rate of human-computer interaction.

[0197] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any robot control method provided in the embodiment of the present application.

[0198] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any robot control method provided in the embodiment of the present application.

[0199] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0200] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0201] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robot control method, characterized in that: include: In response to a user's wake-up command, the robot identifies the user's location and controls the robot to adjust its position so that a visual perception unit on the robot faces the user's location, and collects visual information through the visual perception unit; When a user's task voice command is detected, voice command recognition is started, and before the user's command intention is recognized, human posture recognition is started to perform human posture recognition on each frame image in the visual information continuously collected by the visual perception unit to obtain a human posture recognition result for each frame image; After obtaining the user instruction intention, if it is determined that the user instruction intention indicates that the user has a gesture interaction intention, then according to the front and rear endpoint times of the voice segment corresponding to the task voice instruction, the human body gesture recognition results of each frame image in the visual information within the front and rear endpoint time range are used to determine the final human body gesture recognition result; The robot is controlled to perform an action matching the final human posture recognition result.

2. The method according to claim 1, characterized in that The time of starting the human body posture recognition is synchronized with the time of starting the voice command recognition.

3. The method according to claim 1, characterized in that The process of starting human posture recognition to perform human posture recognition on each frame image in the visual information continuously collected by the visual perception unit to obtain a human posture recognition result for each frame image includes: A sliding window selection is performed on each frame image in the visual information continuously collected by the visual perception unit, and human posture recognition is started for each continuous frame image in each sliding window to determine a human posture recognition result corresponding to each sliding window. The human posture recognition result includes whether each frame image in the sliding window has a set posture action, whether the posture action is stable, and interactive information indicated by the stable posture action.

4. The method according to claim 3, characterized in that: The process of determining the final human posture recognition result by using the human posture recognition result of each frame image in the visual information within the front and rear endpoint time range includes: Determine each target sliding window within the time range of the front and rear endpoints, and a human posture recognition result corresponding to each target sliding window; If there is at least one human posture recognition result corresponding to the target sliding window that contains interactive information indicated by a stable posture action, the final human posture recognition result is determined according to the interactive information indicated by the stable posture action contained in the human posture recognition results corresponding to each target sliding window.

5. The method according to claim 4, characterized in that The process of determining the final human posture recognition result according to the interactive information indicated by the stable posture action contained in the human posture recognition result corresponding to each target sliding window includes: The human posture recognition result of the last target sliding window in the time sequence among the target sliding windows whose human posture recognition results contain the interactive information indicated by the stable posture action is taken as the final human posture recognition result.

6. The method according to claim 4, characterized in that The interactive information indicated by the stable gesture action is the coordinates of the pointing point; and the process of determining the final human gesture recognition result according to the interactive information indicated by the stable gesture action contained in the human gesture recognition result corresponding to each target sliding window includes: According to the pointing point coordinates indicated by the stable gesture actions contained in the human gesture recognition results corresponding to each target sliding window, the mean or median of the pointing point coordinates is calculated as the final human gesture recognition result.

7. The method according to claim 4, characterized in that Also includes: If the human body posture recognition results corresponding to each of the target sliding windows indicate that there is no set posture action, outputting first prompt information, wherein the first prompt information is used to prompt the user to perform the set posture action; If the human posture recognition results corresponding to the target sliding windows do not contain interactive information indicated by a stable posture action, but the human posture recognition result of the target sliding window indicates that the posture action is unstable, then the second prompt information is output, and the second prompt information is used to prompt the user to keep a stable posture.

8. The method according to claim 3, characterized in that The process of starting human posture recognition for each consecutive frame image in each sliding window and determining the human posture recognition result corresponding to each sliding window includes: Initiate human posture recognition for each frame of image in each sliding window to determine whether the image contains a set posture action; If it is determined that the image has a set posture action, further determining whether the set posture action is stable according to each frame image in the sliding window; If it is determined that the set posture action is stable, then the interaction information indicated by the stable posture action is further determined according to each frame image.

9. The method according to claim 1, characterized in that: In response to a user's wake-up command, the process of identifying the user's location includes: The sound source is localized according to the voice segment corresponding to the wake-up command to obtain the direction of the sound source; The visual information of the sound source orientation is collected by the visual perception unit on the robot, and whether the user is recognized is determined based on the visual information. If not, the robot is controlled to rotate along the first rotation direction until the user's position is recognized or one rotation is completed.

10. The method according to any one of claims 1 to 9, characterized in that: The human posture recognition includes hand gesture recognition.

11. A robot control device, characterized in that: include: A wake-up instruction response unit, used to respond to the user's wake-up instruction, identify the user's location and control the robot to adjust the posture so that the visual perception unit on the robot faces the user's location, and collect visual information through the visual perception unit; A human posture recognition unit, used to start voice command recognition when a user's task voice command is detected, and start human posture recognition before recognizing the user's command intention, so as to perform human posture recognition on each frame image in the visual information continuously collected by the visual perception unit, and obtain a human posture recognition result for each frame image; A recognition result backtracking unit is used to determine the final human gesture recognition result based on the human gesture recognition results of each frame image in the visual information within the time range of the front and back endpoints of the voice segment corresponding to the task voice command, if it is determined that the user's instruction intention indicates that the user has an intention of gesture interaction after obtaining the user's instruction intention; The action execution unit is used to control the robot to execute an action matching the final human posture recognition result.

12. A robot, characterized in that: include: Equipment body; A speech unit, a visual perception unit, a travel unit and a control unit are arranged on the device body; The speech unit is used to receive speech data; The visual perception unit is used to collect visual information; The traveling unit is used to adjust the posture of the robot under the control of the control unit; The control unit is used to execute each step of the robot control method described in any one of claims 1-10.

13. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the robot control method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the robot control method according to any one of claims 1 to 10 is implemented.

Images