Robot control method, device, related equipment and computer program product

Abstract

This application discloses a robot control method, apparatus, related equipment, and computer program product. Upon receiving a user's wake-up command, the robot's posture is adjusted so that the visual perception unit faces the user's location, and visual information is collected through the visual perception unit. Upon detecting the user's task voice command, voice command recognition is initiated, and before recognizing the user's command intent, human posture recognition is initiated to perform human posture recognition on the visual information collected by the visual perception unit. Upon detecting the user's intention to interact with a posture, the robot can backtrack the human posture recognition results of each frame image in the visual information within the time range based on the front and back endpoint times of the voice segment corresponding to the task voice command to determine the final human posture recognition result. This application solution can, on the one hand, speed up the response speed, and on the other hand, better conform to the user's usage habits, thereby improving the success rate of human-computer interaction.

Description

Technical Field

[0001] The present application relates to the technical fields of computer vision and human-computer interaction, and more specifically, to a robot control method, apparatus, related equipment, and computer program product. Background Art

[0002] With the development of technology, the use of interactive robots is becoming more and more widespread. Human-computer interaction can be separated from the remote control of remote platforms and allow robots to perform some autonomous behaviors.

[0003] During human-machine interaction, users can use certain body movements (such as gestures) to give voice commands to the machine. For example, a user points to a location on the floor to be cleaned and issues the voice command, "Clean the area I'm pointing to." Traditional approaches typically view the human-machine interaction process as a serial process. This involves first recognizing the user's voice command. After determining the user's intent, the robot adjusts its position so that its camera focuses on the user, captures an image of the user, and performs gesture recognition based on the captured image to determine the interaction information indicated by the user (such as the indicated location). However, user behavior is not always serial. Some users often speak while performing actions, or even perform an action before speaking, and may stop performing the action after the voice command is completed. Because algorithm execution takes time, from the completion of the user's voice command to the machine's recognition of the user's intent, traditional approaches may have already released the gesture by the time gesture recognition begins after determining the user's intent. This results in failure to recognize the gesture and, consequently, interaction failure. Summary of the Invention

[0004] In view of the above problems, this application is proposed to provide a robot control method, apparatus, related equipment and computer program product to improve the success rate of human-computer interaction. The specific solution is as follows:

[0005] In a first aspect, a robot control method is provided, comprising:

[0006] In response to a user's wake-up command, the robot identifies the user's location and controls the robot to adjust its posture so that a visual perception unit on the robot faces the user's location, and collects visual information through the visual perception unit;

[0007] When a user's task voice command is detected, voice command recognition is started, and before the user's command intention is recognized, human posture recognition is started to perform human posture recognition on each frame of the visual information continuously collected by the visual perception unit to obtain a human posture recognition result for each frame of the image;

[0008] After obtaining the user instruction intention, if it is determined that the user instruction intention indicates that the user has a gesture interaction intention, then according to the front and back endpoint times of the voice segment corresponding to the task voice instruction, using the human gesture recognition results of each frame image in the visual information within the front and back endpoint time range, determine a final human gesture recognition result;

[0009] The robot is controlled to perform an action that matches the final human posture recognition result.

[0010] In one possible design, in another implementation of the first aspect of the embodiment of the present application, the time for starting human gesture recognition is synchronized with the time for starting voice command recognition.

[0011] In one possible design, in another implementation of the first aspect of the embodiments of the present application, starting human posture recognition to perform human posture recognition on each frame of visual information continuously collected by the visual perception unit to obtain a human posture recognition result for each frame of image includes:

[0012] A sliding window selection is performed on each frame image in the visual information continuously collected by the visual perception unit, and human posture recognition is initiated for each consecutive frame image in each sliding window to determine the human posture recognition result corresponding to each sliding window. The human posture recognition result includes whether each frame image in the sliding window has a set posture action, whether the posture action is stable, and interactive information indicated by the stable posture action.

[0013] In one possible design, in another implementation of the first aspect of the embodiments of the present application, the process of determining the final human posture recognition result using the human posture recognition results of each frame image in the visual information within the front and rear endpoint time range includes:

[0014] Determine each target sliding window within the front and rear endpoint time range, and a human posture recognition result corresponding to each target sliding window;

[0015] If there is at least one human posture recognition result corresponding to the target sliding window that contains interaction information indicated by a stable posture action, the final human posture recognition result is determined according to the interaction information indicated by the stable posture action contained in the human posture recognition result corresponding to each target sliding window.

[0016] In one possible design, in another implementation of the first aspect of the embodiments of the present application, a process of determining a final human posture recognition result based on interaction information indicated by stable posture actions included in the human posture recognition results corresponding to each target sliding window includes:

[0017] The human posture recognition result of the last target sliding window in the time sequence among the target sliding windows whose human posture recognition results contain interactive information indicated by stable posture movements is taken as the final human posture recognition result.

[0018] In one possible design, in another implementation of the first aspect of the embodiments of the present application, the interaction information indicated by the stable gesture action is the coordinates of the pointing point; and a process of determining a final human gesture recognition result based on the interaction information indicated by the stable gesture action included in the human gesture recognition result corresponding to each target sliding window includes:

[0019] According to the coordinates of the pointing points indicated by the stable gestures contained in the human gesture recognition results corresponding to the target sliding windows, the mean or median of the coordinates of the pointing points is calculated as the final human gesture recognition result.

[0020] In one possible design, in another implementation of the first aspect of the embodiments of the present application, the method further includes:

[0021] If the human posture recognition results corresponding to each of the target sliding windows indicate that the set posture action does not exist, outputting first prompt information, where the first prompt information is used to prompt the user to perform the set posture action;

[0022] If the human posture recognition results corresponding to the target sliding windows do not include interactive information indicated by a stable posture action, but the human posture recognition result of the target sliding window indicates that the posture action is unstable, then a second prompt information is output, and the second prompt information is used to prompt the user to maintain a stable posture.

[0023] In one possible design, in another implementation of the first aspect of the embodiments of the present application, a process of initiating human posture recognition for consecutive image frames within each sliding window and determining a human posture recognition result corresponding to each sliding window includes:

[0024] Initiate human gesture recognition for each frame of image in each sliding window to determine whether the image contains a set gesture action;

[0025] If it is determined that the image has a set posture action, further determining whether the set posture action is stable based on each frame image in the sliding window;

[0026] If it is determined that the set posture action is stable, then the interaction information indicated by the stable posture action is further determined based on each frame image.

[0027] In one possible design, in another implementation of the first aspect of the embodiments of the present application, in response to a wake-up instruction from the user, a process of identifying the location of the user includes:

[0028] Perform sound source localization based on the voice segment corresponding to the wake-up command to obtain the direction of the sound source;

[0029] The visual information of the sound source orientation is collected by the visual perception unit on the robot, and based on the visual information, it is determined whether the user is recognized. If not, the robot is controlled to rotate along the first rotation direction until the user's position is recognized or one rotation is completed.

[0030] In one possible design, in another implementation of the first aspect of the embodiments of the present application, the human posture recognition includes hand gesture recognition.

[0031] In a second aspect, a robot control device is provided, comprising:

[0032] A wake-up command response unit is used to respond to the user's wake-up command, identify the user's location, and control the robot to adjust its posture so that the visual perception unit on the robot faces the user's location, and collect visual information through the visual perception unit;

[0033] a human gesture recognition unit, configured to initiate voice command recognition upon detecting a user's task voice command, and initiate human gesture recognition before recognizing the user's command intention, so as to perform human gesture recognition on each frame of the visual information continuously collected by the visual perception unit, and obtain a human gesture recognition result for each frame of the image;

[0034] a recognition result backtracking unit configured to, after obtaining the user instruction intention, determine a final human gesture recognition result based on the human gesture recognition results of each frame image in the visual information within the time range of the front and back endpoints of the voice segment corresponding to the task voice instruction if it is determined that the user instruction intention indicates that the user has a gesture interaction intention;

[0035] An action execution unit is used to control the robot to execute an action that matches the final human posture recognition result.

[0036] A third aspect provides a robot, comprising:

[0037] Equipment body;

[0038] A speech unit, a visual perception unit, a travel unit, and a control unit are provided on the device body;

[0039] The speech unit is used to receive speech data;

[0040] The visual perception unit is used to collect visual information;

[0041] The travel unit is used to adjust the robot posture under the control of the control unit;

[0042] The control unit is used to execute each step of the robot control method described in any one of the first aspects of the present application.

[0043] In a fourth aspect, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the robot control method described in any one of the first aspects of the present application are implemented.

[0044] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the various steps of the robot control method described in any one of the first aspects of the present application.

[0045] From the above technical solution, it can be seen that the present application does not start human posture recognition processing after recognizing the user's task voice command and obtaining the user's command intention. Instead, upon receiving the user's wake-up command, the robot's position is adjusted so that the visual perception unit is facing the user's position, and visual information is collected through the visual perception unit. On this basis, when the user's task voice command is detected, voice command recognition is started, and before the user's command intention is recognized, human posture recognition is started to perform human posture recognition on each frame of visual information continuously collected by the visual perception unit to obtain a recognition result for each frame of image. After obtaining the user's command intention, if it is determined that the intention indicates that the user has a gesture interaction intention, the human posture recognition results of each frame of image in the visual information within the time range of the preceding and following endpoints of the voice segment corresponding to the task voice command can be backtracked to determine the final human posture recognition result. Obviously, compared with the method of starting visual information collection and human posture recognition after obtaining the user's command intention, the present application solution can recognize human posture in advance, which can not only speed up the response speed, but also better conform to the user's usage habits and improve the success rate of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0047] Figure 1 A schematic diagram of an implementation system architecture of the robot control method provided in an embodiment of the present application;

[0048] Figure 2 A schematic flow chart of a robot control method provided in an embodiment of the present application;

[0049] Figure 3A flowchart of a method for adjusting a robot's posture in response to a wake-up command provided in an embodiment of the present application;

[0050] Figure 4 A flowchart of another method for adjusting a robot's posture in response to a wake-up command provided in an embodiment of the present application;

[0051] Figure 5 A flowchart of another method for adjusting a robot's posture in response to a wake-up command provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the correspondence between a user task voice instruction and an image human posture recognition result provided in an embodiment of the present application;

[0053] Figure 7 This is a schematic structural diagram of a robot control device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0055] Robot interaction solutions are essentially designed to make people's lives more convenient, and more accurate human-machine interaction solutions can provide a better experience. Existing solutions focus primarily on improving the recognition accuracy of related algorithms, such as speech and intent recognition, and image-based recognition of human pointing points.

[0056] The traditional human-computer interaction process typically involves receiving a user's gesture-based voice command (for example, "Clean the area I'm pointing to"), rotating toward the user based on a sound source localization algorithm, then capturing an image of the user and performing gesture recognition (such as gesture recognition) to determine the location the user is pointing at. This serial interaction response process, firstly, increases waiting time after the user issues a voice command. Secondly, it doesn't conform to actual user habits. User interactions aren't always sequential. Some users often point while speaking, or point before speaking. Some even assume the robot is "listening and watching," and drop their gesture immediately or only moments after completing a voice command. Because the voice recognition process takes time, it takes time from the end of the user's voice command to the machine recognizing the user's intent. Therefore, by the time gesture recognition is initiated after the user's intent is recognized, the user may have already dropped their gesture. This misses the user's gesture, resulting in an inability to detect interaction information, such as the coordinates of the gesture, and ultimately, human-computer interaction failure.

[0057] To this end, this application provides a robot control solution. Unlike existing solutions, this application aims to solve the defects of existing solutions, improve the success rate of human-computer interaction, and provide users with a more convenient and comfortable interactive experience by at least improving the human-computer interaction process.

[0058] The robot control solution provided in this application can be applied to control various types of robots. The robot referred to in this application refers to an electronic device that can move autonomously and can be intelligently controlled. In some scenarios, the robot can be a robot with sweeping and cleaning functions. Figure 1 The robot sweeping robot is used as an example to illustrate the structure of the robot control system of the present application, but this does not mean that the robot provided by the present application can only be implemented as a sweeping robot. In other scenarios, the robot of the present application can also be a warehouse logistics robot, such as a freight robot or a goods delivery robot, or the robot of the present application can be a robot waiter, such as a hotel welcome robot, a food serving robot, or a shopping guide robot in a shopping mall.

[0059] The robots listed above are only used for illustrative purposes, and this embodiment includes but is not limited to them.

[0060] The robot control method provided in this application can be applied to Figure 1 As shown in the system architecture, the system may include a robot 100 and a control terminal 200 .

[0061] The robot 100 or the control terminal 200 can be used alone to execute the robot control method provided in the embodiment of the present application. In addition, the robot 100 and the control terminal 200 can also be used in conjunction to execute the robot control method provided in the embodiment of the present application.

[0062] In one possible embodiment, the robot 100 may send data to the control terminal 200 and receive processing results from the control terminal 200. For example, the robot 100 sends the received user's voice command to the control terminal 200. The control terminal 200 recognizes the voice command and returns the recognized user command intent to the robot 100. The robot 100 sends the visual information collected by the visual perception unit to the control terminal 200. The control terminal 200 initiates human posture recognition, obtains human posture recognition results for each frame of image, and returns them to the robot 100. The robot 100 determines the final human posture recognition result based on the returned human posture recognition results and performs the matching action.

[0063] In another possible embodiment, the robot 100 can independently complete the control process, that is, the robot 100 can be locally configured with a voice recognition engine and a human posture recognition engine, and then can locally complete the recognition of voice commands and human postures, and complete the robot's control process based on the recognition results.

[0064] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, usage scenarios, etc. of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0065] The present invention provides a robot control method, which is applied to Figure 1 Taking the computer device in the example as an example, the computer device can be Figure 1 Robot 100 in. Figure 2 , the robot control method specifically includes the following steps:

[0066] Step S100: In response to a user's wake-up instruction, the robot's visual perception unit is directed toward the user and collects visual information.

[0067] Specifically, the user may initiate a wake-up instruction in the form of voice, and the wake-up instruction may be a voice instruction including a set wake-up word.

[0068] After the user initiates the wake-up command, the user's location can be identified through sound source positioning and / or vision-based human recognition, and then the robot can be controlled to adjust its posture so that the visual perception unit on the robot is facing the user's location, and the visual perception unit is started to start collecting visual information.

[0069] The visual perception unit may be composed of at least one camera module or at least one area array ranging module. The camera module may be an RGB camera or other types of camera modules. In one possible implementation, to ensure both recognition and ranging accuracy, the present application may employ at least two camera modules, or at least one camera module in conjunction with an area array ranging module.

[0070] The robot of the present application may be provided with a voice unit, which may be composed of at least one microphone. In some possible implementations, the voice unit may be composed of at least three microphones, so as to estimate the relative direction of the user and the robot based on the time difference between the user's voice command and the three microphones, thereby accelerating the robot's orientation toward the user, or accurately aligning the robot with the speaker issuing the voice command when there are multiple people in the room.

[0071] Compared to existing technologies, which only adjust the robot's posture, activate the camera for image acquisition, and perform human posture recognition after the user issues a specific task voice command and obtains the user's command intent through voice recognition, this application pre-stages the process of adjusting the robot's posture to align with the user and activating the visual perception unit to collect visual information. In other words, the robot's posture adjustment and visual perception unit collection steps are executed immediately after the user issues the wake-up command. This can speed up the response and prepare for the subsequent step of initiating human posture recognition.

[0072] Step S110: When the user's task voice command is detected, voice command recognition is started, and before the user's command intention is recognized, human posture recognition is started to obtain a human posture recognition result corresponding to the visual information.

[0073] Specifically, after waking up the robot, the user can further issue task voice instructions. The task voice instructions can be of various types, for example, task instructions that indicate the user's intention of gesture interaction, such as gesture interaction voice instructions. In addition, they can also be task instructions with other non-gesture interaction intentions, such as voice instructions to order the sweeping robot to return to the base station.

[0074] Considering that the user's task voice instruction may be a task instruction with gesture interaction intention, the user will issue the task voice instruction in conjunction with body movements. In order to avoid the existing solution of performing intent recognition on the task voice instruction and only starting the human body gesture recognition process after obtaining the user's task intention, at which time the user may have stopped the body movement, resulting in the inability to recognize the user's interaction intention, in this step, when the user's task voice instruction is detected, the voice instruction recognition algorithm is first started, that is, the task voice instruction is performed for voice recognition and intent recognition. Secondly, before the voice instruction recognition algorithm processing process obtains the user's instruction intention, the human body gesture recognition process is further started to perform human body gesture recognition on each frame of the visual information continuously collected by the visual perception unit to obtain the human body gesture recognition result for each frame of the image.

[0075] In one possible implementation, human gesture recognition can be initiated synchronously with voice command recognition. That is, upon detecting a user's voice command for a task, both voice command recognition and human gesture recognition are initiated simultaneously. "Synchronous" here can be understood as meaning the same time, or system response time tolerance can be taken into account, ensuring that both recognition processes are initiated within a very short timeframe.

[0076] This step changes the existing method of treating voice command recognition and human posture recognition as a serial processing flow, and advances the human posture recognition process (compared to the existing method of starting the human posture recognition process only after voice command recognition obtains the user's command intention). This is more in line with the user's usage habits, that is, issuing voice commands while performing actions, which improves the interaction success rate.

[0077] In this step, human posture recognition is started to perform human posture recognition on each frame image in the visual information continuously collected by the visual perception unit to obtain a human posture recognition result for each frame image.

[0078] The gestures to be recognized by human posture recognition can be determined based on the interaction actions specified by the business logic. For example, if the pre-agreed human-computer interaction action is a hand gesture, then the gestures to be recognized by human posture recognition here are the pre-defined gesture types. In addition, human-computer interaction actions can also include movements of other parts of the body, such as leg movements, toe movements, head movements, etc. The subsequent embodiments of this application only use hand gestures as an example for explanation.

[0079] Step S120: When it is determined that the user has gesture interaction intention, the human gesture recognition results of each frame image within the time range are backtracked according to the front and back endpoint times of the voice segment corresponding to the task voice instruction to determine the final human gesture recognition result.

[0080] Specifically, voice command recognition is started in the previous step to identify the user command intention corresponding to the user's task voice command. After identifying the user command intention, if it is further determined that the user command intention represents the user's intention of gesture interaction, then the front and back endpoint times of the voice segment corresponding to the task voice command can be determined at this time, and then the human posture recognition results of each frame image within the front and back endpoint time range can be used to determine the final human posture recognition result.

[0081] The results of voice command recognition can include the front-end and back-end points of the speech, the corresponding text information, and the user's command intent. When the front-end point of the speech is recognized, the time of the front-end point can be recorded and the voice segment can be retained. When the back-end point of the speech is recognized, the time of the back-end point can be recorded and the voice segment capture can be terminated. The captured voice segment is fed into the speech recognition model to obtain the recognized text corresponding to the voice task command. This recognized text is then fed into the intent recognition model to determine the user's command intent.

[0082] Among them, the speech recognition model and the intent recognition model can adopt the currently existing models, or can adopt models with speech recognition and intent recognition functions developed in the future. This case does not limit this.

[0083] Step S130: Control the robot to perform an action that matches the final human posture recognition result.

[0084] Specifically, after determining the final human posture recognition result in the above steps, the interaction information indicated by the user's gesture can be obtained. For example, in the case of hand gestures, the position coordinates indicated by the user's gesture can be obtained. Based on this, the robot can be controlled to perform an action that matches the final human posture recognition result.

[0085] For example, a user interacts with a robot vacuum through gestures. For example, the user points to the ground and issues a voice command, "Clean the area I'm pointing to." After obtaining the final human gesture recognition results using the method in this embodiment, the coordinates of the ground location pointed to by the user's gesture can be determined. The robot can then be controlled to move to that location and perform the cleaning task.

[0086] The robot control method provided in the embodiments of the present application does not begin human posture recognition processing only after recognizing the user's task voice command and obtaining the user's command intent. Instead, upon receiving the user's wake-up command, the robot's position is adjusted so that the visual perception unit faces the user's position and begins collecting visual information through the visual perception unit. On this basis, voice command recognition is initiated when the user's task voice command is detected, and human posture recognition is initiated before the user's command intent is recognized. Human posture recognition is performed on each frame of visual information continuously collected by the visual perception unit to obtain a recognition result for each frame of image. After obtaining the user's command intent, if it is determined that the intention indicates the user's intention for gesture interaction, the human posture recognition results of each frame of visual information within the time range of the preceding and following endpoints of the voice segment corresponding to the task voice command are backtracked to determine the final human posture recognition result. Obviously, compared with the method of only starting visual information collection and human posture recognition after obtaining the user's command intent, the present application solution can perform human posture recognition in advance, which can not only speed up the response speed, but also better conform to the user's usage habits and improve the success rate of human-computer interaction.

[0087] In some embodiments of the present application, the process of identifying the user's location in response to the user's wake-up instruction in the aforementioned step S100 is described.

[0088] In this embodiment, a sound source localization algorithm may be used to identify the user's location, or a vision-based human figure recognition algorithm may be used to identify the user's location. Alternatively, the sound source localization algorithm and the human figure recognition algorithm may be combined to identify the user's location.

[0089] A possible implementation method is to combine Figure 3 As shown, the following steps may be included:

[0090] Step S11: localize the sound source according to the voice segment corresponding to the wake-up command to obtain the direction of the sound source.

[0091] Step S12: The visual perception unit on the robot collects visual information of the sound source orientation, and determines whether the user is recognized based on the visual information.

[0092] Here, a human figure recognition algorithm can be used to detect whether a user is recognized based on visual information. This algorithm can utilize a pre-trained deep learning model. The visual information can be a planar image (such as an RGB image, infrared image, or grayscale image) or a point cloud image obtained using an area array ranging module.

[0093] If the user is not recognized based on the visual information, step S13 may be executed; if the user is recognized based on the visual information, step S14 may be executed.

[0094] Step S13: Control the robot to rotate in place along the first rotation direction. During the rotation process, the robot continues to detect whether the user is recognized based on the collected visual information until the user's position is recognized or the robot returns to its original position after rotating one circle.

[0095] Step S14: Get the user's location.

[0096] Another possible implementation is to combine Figure 4 As shown, the following steps may be included:

[0097] Step S21: localize the sound source according to the voice segment corresponding to the wake-up command to obtain a first angle of the sound source relative to the robot.

[0098] Specifically, the process of sound source localization is to estimate the relative direction of the sound source to the robot by using the time difference between the voice segment corresponding to the wake-up command reaching different microphones in the microphone array, and then obtain the first angle of the relative direction relative to the robot's orientation.

[0099] Step S22: Determine whether the first threshold range of the first angle is within the field of view of the robot's visual perception unit to determine whether the robot's posture orientation needs to be rotated.

[0100] In one possible implementation, based on the calibration information of the robot's visual perception unit's installation location on the robot, the angle of the human figure in the image captured by the visual perception unit can be converted to the robot coordinate system. This means that the relative orientation of the human figure in the image relative to the robot can be determined. Alternatively, the X-axis of the robot coordinate system can be oriented toward the forward direction of the robot's travel unit. The visual perception unit is typically mounted directly in front of the robot, facing the X-axis of the robot coordinate system. The angle of the human figure in the image is equivalent to the angle along the X-axis of the robot coordinate system.

[0101] The first threshold range can be pre-set, for example, to [-22.5°, 22.5°]. It's understood that a smaller first threshold range places higher demands on the sound source localization algorithm, but it can also help prevent the robot from erroneously targeting a humanoid when multiple people are present.

[0102] In this step, if it is determined that the first threshold range of the first angle is within the field of view of the robot's visual perception unit, it can be determined that there is no need to rotate the robot's posture orientation. Otherwise, in order to ensure that the visual perception unit can capture images in the first angle direction, the robot's posture orientation needs to be rotated.

[0103] If the result of the judgment is that rotation is required, step S23 may be executed; otherwise, step S24 may be executed.

[0104] Step S23: determine a first rotation direction, and rotate the robot along the first rotation direction until the visual perception unit faces a first angle.

[0105] The first rotation direction may be clockwise or counterclockwise.

[0106] In a possible implementation, the first rotation direction may be a direction that causes the robot to rotate to a direction with a shorter travel direction facing the first angle.

[0107] Step S24: Determine whether there is a human figure within the first threshold range of the first angle. If not, proceed to step S25. If so, proceed to step S26.

[0108] It is understandable that in actual application scenarios, there may not be a humanoid user within the first threshold range of the first angle where the sound source is located, as obtained by the sound source localization algorithm. For example, when the background noise or the robot's own noise is large, the sound source localization algorithm may be interfered with by the noise, resulting in a certain deviation in the result, and no user exists within the first threshold range of the first angle. At this time, in an optional processing method, step S25 can be executed. Of course, it is also possible to remain motionless and wait for subsequent instructions from the user. If it is determined that there is a human figure within the first threshold range of the first angle, it means that the user's image can be collected at present. At this time, the user's location can be obtained, and the current posture of the robot can be kept unchanged, or step S26 can be executed.

[0109] Step S25: Continue rotating along the first rotation direction to determine whether there is a human figure in the field of view of the robot's visual perception unit until the robot faces the first angle again.

[0110] If a human figure is found during the rotation process, the user's location can be obtained, and the robot's current posture can be kept unchanged, or step S26 can be executed. If no human figure is found during the rotation process, step S27 can be executed.

[0111] Step S26: Rotate the robot until the visual perception unit faces the direction of the human figure.

[0112] Specifically, in order to improve the quality of the image captured by the visual perception unit, the robot can be rotated so that the visual perception unit faces the direction of the human figure.

[0113] Step S27: Wait for subsequent instructions from the user.

[0114] Specifically, if the user's location is not detected after all the above steps, the system may continue to wait for subsequent instructions from the user.

[0115] Another possible implementation is that when the robot is working in a specific mode, it can be assumed that the sound source positioning angle is inaccurate. For example, when the robot is in the maximum suction mode, due to the large self-noise of the robot, it can be assumed that the result obtained by using the sound source positioning is inaccurate. Alternatively, when the robot is not equipped with more than three microphones, the sound source positioning algorithm cannot be used. In this embodiment, only the human figure recognition algorithm based on vision can be used to identify the user's location, combined with the human figure recognition algorithm based on vision. Figure 5 As shown, the following steps may be specifically included:

[0116] Step S31: respond to the wake-up instruction and start human figure recognition.

[0117] Step S32: Determine whether there is a human figure in the field of view of the robot's visual perception unit. If so, the user's location can be obtained and the robot's current position can be kept unchanged, or step S33 can be executed. If no human figure is found, step S34 can be executed.

[0118] Step S33: Rotate the robot until the visual perception unit faces the direction of the human figure.

[0119] Specifically, in order to improve the quality of the image captured by the visual perception unit, the robot can be rotated so that the visual perception unit faces the direction of the human figure.

[0120] Step S34: Continue rotating along the second rotation direction to determine whether there is a human figure in the field of view of the robot's visual perception unit until the robot returns to the starting position again.

[0121] If a human figure is found during the rotation, the user's location can be obtained, and the robot's current posture can be kept unchanged, or step S33 can be executed. If no human figure is found during the robot's 360-degree rotation and return to the starting posture, step S35 can be executed.

[0122] The second rotation direction may be clockwise or counterclockwise.

[0123] Step S35: Wait for subsequent instructions from the user.

[0124] Specifically, if the user's location is not detected after all the above steps, the system may continue to wait for subsequent instructions from the user.

[0125] The above embodiments introduce several implementation methods of identifying the user's location in response to the user's wake-up command, and controlling the robot to adjust the posture so that the visual perception unit is facing the user's location. Any one of the methods can be used in actual applications, or other variations can be used. This application will not explain them one by one.

[0126] In some embodiments of the present application, the process of starting human posture recognition and obtaining a human posture recognition result corresponding to visual information in the aforementioned step S110 is described.

[0127] In a possible implementation, human posture recognition may be performed on each image frame in the visual information continuously collected by the visual perception unit to obtain a human posture recognition result for each image frame.

[0128] Combine Figure 6 As shown, in another possible implementation, this embodiment can perform sliding window selection on each frame image in the visual information continuously collected by the visual perception unit, and start human posture recognition on each consecutive frame image in each sliding window to determine the human posture recognition result corresponding to each sliding window.

[0129] When selecting each frame image in the visual information through sliding window, the selection can be made according to the set window length and sliding step length. The window length is greater than or equal to 2 frames of image ( Figure 6 In the example above, the window length is 2. In a specific implementation, taking a 25 fps visual perception unit as an example, the window length of the sliding window can be set to 20 frames. The sliding step size can be equal to the window length, that is, the sliding windows do not need to overlap.

[0130] When performing human posture recognition on each frame image within each sliding window, it is possible to determine whether the image within the sliding window has a set posture action (referred to as whether there is action). If the image has a set posture action, it is possible to further determine whether the posture action is stable based on each frame image within the sliding window.

[0131] Specifically, to determine whether the posture and movement of each frame within the sliding window are stable, bone node detection technology can be used to extract the coordinates of multiple joints on the limb in image space. The coordinates of the same joint in image space are then compared between consecutive frames to see if the position change of the same joint in image space exceeds a set distance threshold. If so, the posture and movement are considered unstable; otherwise, they are considered stable. Alternatively, image time series modeling can be used to predict the stability of the posture and movement of each frame within the sliding window using deep learning algorithms.

[0132] Furthermore, when it is determined that the posture and action of each frame image in the sliding window is stable, the interactive information indicated by the stable posture and action can also be determined based on each frame image in the difference.

[0133] In different interactive tasks, the interactive information indicated by gestures may be different. For example, gestures can indicate location information, such as indicating the ground area that the sweeping robot needs to clean; indicate operation instructions, such as instructing the robot to grab, put down, stop, start, confirm and cancel operation instructions; indicate state control information, such as instructing the robot to change the operating speed, adjust the working mode, etc.

[0134] In this embodiment, the process of determining the position information indicated by the stable posture action is described by taking the interaction information as the indicated position information as an example.

[0135] In one optional implementation, multiple joint points can be extracted from each image frame within a sliding window and mapped to physical space. For example, the positions of the eyes and fingers can be extracted and mapped to physical space. A straight line passing through the eyes and fingers is calculated, and the intersection of this line and the ground is calculated to obtain the coordinates of the pointing point. The pointing point coordinates are calculated for each image frame within the sliding window, and the average or median of the coordinates is taken as the coordinates of the pointing point corresponding to the sliding window.

[0136] Another alternative implementation involves pre-training a pointing point prediction model using a deep learning algorithm. Each image frame within the sliding window is then fed into the model to obtain the model-predicted pointing point coordinates for that image. Furthermore, the average or median of the pointing point coordinates for each frame is taken as the pointing point coordinate for the sliding window.

[0137] The method introduced in this embodiment performs sliding window selection on the visual information collected by the visual perception unit, divides each continuous frame image into several sliding windows, and then identifies the human posture recognition result corresponding to each sliding window according to the sliding windows, which can reduce the error of the single-frame image recognition result.

[0138] On this basis, the process of step S120, which is to trace back the human posture recognition results of each frame image within the time range according to the front and back endpoint times of the voice segment corresponding to the task voice instruction, and determine the final human posture recognition result, is introduced.

[0139] Combine Figure 6 As shown in FIG, according to the front and back endpoints of the voice segment corresponding to the task voice command, each target sliding window within the time range of the front and back endpoints can be located, as well as the human posture recognition result corresponding to each target sliding window.

[0140] If there is at least one target sliding window corresponding to the human posture recognition result that contains interaction information indicated by a stable posture action, the final human posture recognition result can be determined based on the interaction information indicated by the stable posture action contained in the human posture recognition results corresponding to each target sliding window.

[0141] If the time of the start frame of a sliding window is before the front end point and the time of the last frame is after the front end point, the sliding window can be processed as a target sliding window or not. Similarly, if the time of the start frame of a sliding window is before the back end point and the time of the last frame is after the back end point, the sliding window can be processed as a target sliding window or not. In the following embodiments of the present application, the above two types of sliding windows can be processed as target sliding windows.

[0142] In one possible implementation, the process of determining the final human posture recognition result based on the interactive information indicated by the stable posture movements contained in the human posture recognition results corresponding to each target sliding window can be to take the human posture recognition result of the last target sliding window in the time sequence among the target sliding windows whose human posture recognition results contain interactive information indicated by the stable posture movements as the final human posture recognition result.

[0143] In addition, the interaction information indicated by the stable posture movements in each target sliding window can be comprehensively considered, and the final human posture recognition result, that is, the final interaction information, can be calculated according to the set fusion strategy.

[0144] Take the interactive information as the coordinates of the pointing point as an example to illustrate:

[0145] In this application, the mean or median of the pointing point coordinates indicated by the stable posture movements contained in the human posture recognition results corresponding to each target sliding window can be calculated as the final human posture recognition result.

[0146] Combine Figure 6 As shown, there are multiple target windows within the time range of the front end point and the back end point. The windows are numbered from left to right as Window 1, Window 2, Window 3, Window 4, Window 5, Window 6, and Window 7. Windows 2-6 are target windows.

[0147] The user corresponding to sliding window 1 has not started to take any action.

[0148] Slide window 2 and slide window 3 belong to the process of the user raising his hand. There is gesture action, but the action is unstable.

[0149] There is a gesture on the sliding window 4, and the motion is stable, and the pointing point P0 corresponding to the gesture can be calculated.

[0150] There is a gesture on the sliding window 5, and the motion is stable, and the pointing point P1 corresponding to the gesture can be calculated.

[0151] Sliding window 6 belongs to the process of the user lowering his arm. There is a gesture action, but the action is unstable.

[0152] Slider window 7 indicates that the user has already released the gesture, making it undetectable. Furthermore, the final endpoint of the voice signal indicates that the user stops performing the gesture shortly after speaking the voice command. Traditional approaches, where the user has already released the gesture by the time voice recognition determines the user's command intent, may result in the gesture not being detected.

[0153] According to the solution of the present application, it can be determined that two target sliding windows contain interactive information (i.e., pointing points) indicated by stable actions. Based on the interactive information indicated by the stable actions contained in the two target sliding windows, the final human posture recognition result can be determined, that is, the final pointing point is determined based on the pointing point P0 corresponding to sliding window 4 and the pointing point P1 corresponding to sliding window 5.

[0154] For example, the pointing point P1 corresponding to the sliding window 5 that is later in time sequence is taken as the final pointing point. Alternatively, the mean or median of the pointing points P0 and P1 can be taken as the final pointing point.

[0155] It should be noted that in the above steps, based on the front and back endpoints of the voice segment corresponding to the task voice command, each target sliding window within the time range of the front and back endpoints is located, and based on the human posture recognition result corresponding to each target sliding window:

[0156] If the human posture recognition results corresponding to each target sliding window indicate that the set posture action is not present, it can be determined that the user has not performed the set posture action. In this case, a first prompt message can be output to prompt the user to perform the set posture action. For example, in the finger-sweep control process of a sweeping robot, the first prompt message can be "Please point to the location to be cleaned with your hand."

[0157] In another case, if it is determined that none of the human gesture recognition results corresponding to each target sliding window contain interaction information indicated by a stable gesture, but the human gesture recognition results for the target sliding window indicate an unstable gesture, it can be assumed that the user performed the designated gesture, but the movement process was unstable, making it impossible to calculate accurate interaction information based on this unstable gesture. Therefore, in this case, a second prompt message can be output to prompt the user to maintain a stable posture. For example, if the user's gesture is determined to be unstable during the process of indicating a cleaning location through gesture, a second prompt message such as "Please maintain the current gesture stable" can be output.

[0158] The robot control methods provided in the above-mentioned embodiments of the present application advance the process of human gesture recognition compared to traditional solutions. For example, human gesture recognition is performed simultaneously with the recognition of the user's task voice command. In this way, after the user's intention is obtained through voice recognition, if it is determined that the intention indicates the presence of gesture interaction intention, the human gesture recognition results of each frame image within the time range can be traced back according to the front and back endpoint times of the voice segment corresponding to the task voice command to determine the final human gesture recognition result. The use of the present application solution can improve the response speed of human-computer interaction and increase the success rate of human-computer interaction.

[0159] The robot control device provided in an embodiment of the present application is described below. The robot control device described below and the robot control method described above can be referenced to each other.

[0160] See also Figure 7 , Figure 7 This is a schematic structural diagram of a robot control device disclosed in an embodiment of the present application.

[0161] like Figure 7 As shown, the device may include:

[0162] A wake-up instruction response unit 11 is used to respond to the user's wake-up instruction, identify the user's location, and control the robot to adjust its posture so that the visual perception unit on the robot faces the user's location, and collect visual information through the visual perception unit;

[0163] A human gesture recognition unit 12 is configured to initiate voice command recognition upon detecting a user's task voice command, and initiate human gesture recognition before recognizing the user's command intention, so as to perform human gesture recognition on each frame of the visual information continuously collected by the visual perception unit, and obtain a human gesture recognition result for each frame of the image;

[0164] The recognition result backtracking unit 13 is configured to, after obtaining the user instruction intention, determine a final human gesture recognition result based on the human gesture recognition results of each frame image in the visual information within the time range of the front and back endpoints of the voice segment corresponding to the task voice instruction if it is determined that the user instruction intention indicates that the user has a gesture interaction intention;

[0165] The action execution unit 14 is used to control the robot to execute an action that matches the final human posture recognition result.

[0166] In a possible implementation, the time of starting the human gesture recognition is synchronized with the time of starting the voice command recognition.

[0167] In one possible implementation, the human posture recognition unit initiates human posture recognition to perform human posture recognition on each frame of the visual information continuously collected by the visual perception unit, and obtains a human posture recognition result for each frame of the image, including:

[0168] A sliding window selection is performed on each frame image in the visual information continuously collected by the visual perception unit, and human posture recognition is initiated for each consecutive frame image in each sliding window to determine the human posture recognition result corresponding to each sliding window. The human posture recognition result includes whether each frame image in the sliding window has a set posture action, whether the posture action is stable, and interactive information indicated by the stable posture action.

[0169] In one possible implementation, the recognition result backtracking unit determines the final human posture recognition result by using the human posture recognition results of each frame image in the visual information within the front and back endpoint time range, including:

[0170] Determine each target sliding window within the front and rear endpoint time range, and a human posture recognition result corresponding to each target sliding window;

[0171] If there is at least one human posture recognition result corresponding to the target sliding window that contains interaction information indicated by a stable posture action, the final human posture recognition result is determined according to the interaction information indicated by the stable posture action contained in the human posture recognition result corresponding to each target sliding window.

[0172] In one possible implementation, the recognition result backtracking unit determines the final human posture recognition result based on the interactive information indicated by the stable posture actions included in the human posture recognition result corresponding to each target sliding window, including:

[0173] The human posture recognition result of the last target sliding window in the time sequence among the target sliding windows whose human posture recognition results contain interactive information indicated by stable posture movements is taken as the final human posture recognition result.

[0174] In another possible implementation, the interaction information indicated by the stable gesture is the coordinates of the pointing point; and the recognition result backtracking unit determines the final human gesture recognition result based on the interaction information indicated by the stable gesture included in the human gesture recognition result corresponding to each target sliding window, including:

[0175] According to the coordinates of the pointing points indicated by the stable gestures contained in the human gesture recognition results corresponding to the target sliding windows, the mean or median of the coordinates of the pointing points is calculated as the final human gesture recognition result.

[0176] In one possible implementation, the apparatus of the present application may further include:

[0177] a prompt output unit configured to output first prompt information for prompting a user to perform a set gesture if it is determined that the human gesture recognition results corresponding to each of the target sliding windows indicate that no set gesture action exists, based on the final human gesture recognition result obtained by the recognition result backtracking unit;

[0178] If it is determined that none of the human posture recognition results corresponding to the target sliding windows contain interactive information indicated by a stable posture action, but the human posture recognition result of the target sliding window indicates that the posture action is unstable, then second prompt information is output, and the second prompt information is used to prompt the user to maintain a stable posture.

[0179] In one possible implementation, the human posture recognition unit initiates human posture recognition for each consecutive frame image in each sliding window, and determines a human posture recognition result corresponding to each sliding window, including:

[0180] Initiate human gesture recognition for each frame of image in each sliding window to determine whether the image contains a set gesture action;

[0181] If it is determined that the image has a set posture action, further determining whether the set posture action is stable based on each frame image in the sliding window;

[0182] If it is determined that the set posture action is stable, then the interaction information indicated by the stable posture action is further determined based on each frame image.

[0183] In one possible implementation, the process of the wake-up instruction response unit identifying the user's location in response to the user's wake-up instruction includes:

[0184] Perform sound source localization based on the voice segment corresponding to the wake-up command to obtain the direction of the sound source;

[0185] The visual information of the sound source orientation is collected by the visual perception unit on the robot, and based on the visual information, it is determined whether the user is recognized. If not, the robot is controlled to rotate along the first rotation direction until the user's position is recognized or one rotation is completed.

[0186] In a possible implementation, the human posture recognition includes hand gesture recognition.

[0187] In some embodiments of the present application, a robot is further provided, comprising:

[0188] Equipment body;

[0189] A speech unit, a visual perception unit, a travel unit, and a control unit are provided on the device body;

[0190] The voice unit is used to receive voice data;

[0191] The visual perception unit is used to collect visual information;

[0192] The travel unit is used to adjust the posture of the robot under the control of the control unit;

[0193] The control unit is used to implement each step of the robot control method of any of the aforementioned embodiments.

[0194] The visual perception unit may be composed of at least one camera module or at least one area array ranging module. The camera module may be an RGB camera or other types of camera modules. In one possible implementation, to ensure both recognition and ranging accuracy, the present application may employ at least two camera modules, or at least one camera module in conjunction with an area array ranging module.

[0195] The speech unit can be composed of at least one microphone. In some possible implementations, the speech unit can be composed of at least three microphones. This allows the robot to estimate the relative direction of the user and the robot based on the time difference between the user's voice command and the three microphones, accelerating the robot's orientation toward the user, or accurately aligning the robot with the speaker issuing the voice command when there are multiple people in the room.

[0196] The robot provided by the present application, wherein the control unit can execute the robot control method introduced in the aforementioned embodiment, can speed up the response speed, and on the other hand is more in line with the user's usage habits, thereby improving the success rate of human-computer interaction.

[0197] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any robot control method provided in the embodiment of the present application.

[0198] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any robot control method provided in the embodiment of the present application.

[0199] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0200] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referenced to each other.

[0201] The above description of the disclosed embodiments will enable those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A robot control method, characterized in that: include: In response to a user's wake-up command, the robot identifies the user's location and controls the robot to adjust its posture so that a visual perception unit on the robot faces the user's location, and collects visual information through the visual perception unit; When a user's task voice command is detected, voice command recognition is started, and before the user's command intention is recognized, human posture recognition is started to perform human posture recognition on each frame of the visual information continuously collected by the visual perception unit to obtain a human posture recognition result for each frame of the image; After obtaining the user instruction intention, if it is determined that the user instruction intention indicates that the user has a gesture interaction intention, then according to the front and back endpoint times of the voice segment corresponding to the task voice instruction, using the human gesture recognition results of each frame image in the visual information within the front and back endpoint time range, determine a final human gesture recognition result; Controlling the robot to perform an action matching the final human posture recognition result; The process of starting human posture recognition to perform human posture recognition on each frame of the visual information continuously collected by the visual perception unit to obtain a human posture recognition result for each frame of the image includes: A sliding window selection is performed on each frame image in the visual information continuously collected by the visual perception unit, and human posture recognition is initiated for each consecutive frame image in each sliding window to determine the human posture recognition result corresponding to each sliding window. The human posture recognition result includes whether each frame image in the sliding window has a set posture action, whether the posture action is stable, and interactive information indicated by the stable posture action.

2. The method according to claim 1, characterized in that The time of starting the human body posture recognition is synchronized with the time of starting the voice command recognition.

3. The method according to claim 1, characterized in that The process of determining a final human posture recognition result by using the human posture recognition results of each frame image in the visual information within the front and rear endpoint time range includes: Determine each target sliding window within the front and rear endpoint time range, and a human posture recognition result corresponding to each target sliding window; If there is at least one human posture recognition result corresponding to the target sliding window that contains interaction information indicated by a stable posture action, the final human posture recognition result is determined according to the interaction information indicated by the stable posture action contained in the human posture recognition result corresponding to each target sliding window.

4. The method according to claim 3, characterized in that The process of determining the final human posture recognition result based on the interactive information indicated by the stable posture action contained in the human posture recognition result corresponding to each target sliding window includes: The human posture recognition result of the last target sliding window in the time sequence among the target sliding windows whose human posture recognition results contain interactive information indicated by stable posture movements is taken as the final human posture recognition result.

5. The method according to claim 3, characterized in that The interactive information indicated by the stable gesture action is the coordinates of the pointing point; and a process of determining a final human gesture recognition result based on the interactive information indicated by the stable gesture action contained in the human gesture recognition result corresponding to each target sliding window includes: According to the coordinates of the pointing points indicated by the stable gestures contained in the human gesture recognition results corresponding to the target sliding windows, the mean or median of the coordinates of the pointing points is calculated as the final human gesture recognition result.

6. The method according to claim 3, characterized in that Also includes: If the human posture recognition results corresponding to each of the target sliding windows indicate that the set posture action does not exist, outputting first prompt information, where the first prompt information is used to prompt the user to perform the set posture action; If the human posture recognition results corresponding to the target sliding windows do not include interactive information indicated by a stable posture action, but the human posture recognition result of the target sliding window indicates that the posture action is unstable, then a second prompt information is output, and the second prompt information is used to prompt the user to maintain a stable posture.

7. The method according to claim 1, characterized in that The process of starting human posture recognition for each consecutive frame image in each sliding window and determining the human posture recognition result corresponding to each sliding window includes: Initiate human gesture recognition for each frame of image in each sliding window to determine whether the image contains a set gesture action; If it is determined that the image has a set posture action, further determining whether the set posture action is stable based on each frame image in the sliding window; If it is determined that the set posture action is stable, then the interaction information indicated by the stable posture action is further determined based on each frame image.

8. The method according to claim 1, characterized in that In response to a user's wake-up command, the process of identifying the user's location includes: Perform sound source localization based on the voice segment corresponding to the wake-up command to obtain the direction of the sound source; The visual information of the sound source orientation is collected by the visual perception unit on the robot, and based on the visual information, it is determined whether the user is recognized. If not, the robot is controlled to rotate along the first rotation direction until the user's position is recognized or one rotation is completed.

9. The method according to any one of claims 1 to 8, characterized in that The human posture recognition includes hand gesture recognition.

10. A robot control device, characterized in that: include: A wake-up command response unit is used to respond to the user's wake-up command, identify the user's location, and control the robot to adjust its posture so that the visual perception unit on the robot faces the user's location, and collect visual information through the visual perception unit; A human posture recognition unit is configured to initiate voice command recognition upon detecting a user's task voice command, and before recognizing the user's command intention, initiate human posture recognition to perform human posture recognition on each frame of the visual information continuously collected by the visual perception unit to obtain a human posture recognition result for each frame of the visual information, wherein the process of initiating human posture recognition to perform human posture recognition on each frame of the visual information continuously collected by the visual perception unit to obtain a human posture recognition result for each frame of the visual information comprises: performing sliding window selection on each frame of the visual information continuously collected by the visual perception unit, initiating human posture recognition on each consecutive frame of the visual information within each sliding window, and determining a human posture recognition result corresponding to each sliding window, wherein the human posture recognition result includes whether each frame of the visual information within the sliding window has a set posture action, whether the posture action is stable, and interactive information indicated by the stable posture action; a recognition result backtracking unit configured to, after obtaining the user instruction intention, determine a final human gesture recognition result based on the human gesture recognition results of each frame image in the visual information within the time range of the front and back endpoints of the voice segment corresponding to the task voice instruction if it is determined that the user instruction intention indicates that the user has a gesture interaction intention; An action execution unit is used to control the robot to execute an action that matches the final human posture recognition result.

11. A robot, characterized in that: include: Equipment body; A speech unit, a visual perception unit, a travel unit, and a control unit are provided on the device body; The speech unit is used to receive speech data; The visual perception unit is used to collect visual information; The traveling unit is used to adjust the posture of the robot under the control of the control unit; The control unit is used to execute each step of the robot control method according to any one of claims 1 to 9.

12. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the robot control method according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the robot control method according to any one of claims 1 to 9 is implemented.

Images