Robots and their interactive control methods, devices, and storage media

Abstract

This application discloses a robot and its interactive control method, device, and storage medium. Upon receiving a voice command, this application calculates the probability distribution of the user's voice source originating from multiple directions using a sound source localization method. Candidate directions are selected sequentially according to decreasing probability confidence. After each selection, the robot is controlled to turn in that direction and acquires an image from its image sensor. Based on the image, it determines whether the target user has been identified. If so, the selection of candidate directions stops; otherwise, the next candidate direction is selected until the target user is identified or a set turning cutoff condition is met. This application's approach, by selecting different candidate directions sequentially according to probability confidence and continuing to select the next candidate direction when the target user is not identified in one direction, increases the probability of locating the target user, thus improving the success rate of user interaction.

Description

Technical Field

[0001] This application relates to the field of intelligent devices and their control technology, and more specifically, to a robot and its interactive control method, device, and storage medium. Background Technology

[0002] With the development of society, economy, and science and technology, people's pursuit of a high standard of living is growing. Against this backdrop, intelligent robots are increasingly appearing in the public eye. Through intelligent voice, machine vision, and other human-computer interaction methods, people can operate machines more easily than before, significantly improving the convenience and comfort of life and work. Taking home service robots as an example, as an important member of the smart home ecosystem, they have become increasingly popular with consumers in recent years.

[0003] Existing robots are generally equipped with microphone arrays and image sensors, which can recognize the voice commands given by the user, estimate the user's position by locating the sound source, turn to the user's position, and recognize the user's actions through images, thereby controlling the robot to perform specific tasks.

[0004] Existing sound source localization methods generally estimate the user's position based on the time difference of sound arrival at different microphones in a microphone array. When the robot's working environment is complex, its work area may contain various obstacles. During movement, the robot may approach these obstacles (including walls), causing sound signals to be reflected by the obstacle surfaces, leading to incorrect user location by existing sound source localization methods. Furthermore, various other sound interference sources may exist in the working environment, such as audio played on television or radio, which can also easily cause existing sound source localization methods to mislocate the user. When existing sound source localization methods fail, the robot may incorrectly turn in a direction other than the user's location, resulting in the inability to identify the target user and ultimately, user interaction failure. Summary of the Invention

[0005] In view of the above problems, this application is proposed to provide a robot and its interactive control method, device, and storage medium to avoid direct failure of user interaction when the sound source is mislocated, thereby improving the success rate of user interaction. The specific solution is as follows:

[0006] Firstly, a robot interaction control method is provided, wherein the robot is equipped with a microphone array and an image sensor, and the interaction control method includes:

[0007] Upon receiving a voice command, the system calculates the probability distribution of the user's voice source originating from multiple directions using the microphone array and a sound source localization method.

[0008] According to the probability distribution, candidate directions are selected sequentially in descending order of probability confidence. After each candidate direction is selected, the robot is controlled to turn to the candidate direction and acquire the image collected by the image sensor.

[0009] Based on the image, determine whether the target user has been identified. If the target user is identified, stop selecting candidate directions. If the target user is not identified, continue selecting the next candidate direction until the target user is identified or the set turning cutoff condition is met.

[0010] Preferably, determining whether a target user has been identified based on the image includes:

[0011] The system detects whether the image contains a human body. If it does, the target user is identified; otherwise, the target user is not identified.

[0012] or,

[0013] The system detects whether the image contains a human body. If not, it determines that the target user has not been identified. If so, it determines whether the identity of the human body in the image matches the identity of the user who issued the voice command. If they match, it determines that the target user has been identified. If they do not match, it determines that the target user has not been identified.

[0014] Preferably, candidate directions are selected sequentially according to the probability distribution and in descending order of probability confidence, including:

[0015] Each time a candidate direction is selected, among the candidate directions that are not yet covered by the viewpoint, one of the top N candidate directions is selected according to the probability confidence level from largest to smallest. The candidate directions that are not yet covered by the viewpoint are: in the plane where the robot is located, the candidate directions that are excluded from all candidate directions covered by the horizontal viewpoint of the image sensor when the robot is in each of the selected candidate directions, N≥1.

[0016] Preferably, candidate directions are selected sequentially according to the probability distribution and in descending order of probability confidence, including:

[0017] Each time a candidate direction is selected, among the candidate directions that are not yet covered by the viewpoint, the horizontal viewpoint range of the image sensor is used as the window length, and a set step angle is used as the sliding step to obtain a set of candidate directions. The set step angle is smaller than the window length.

[0018] The total probability confidence of each candidate direction set is obtained by summing the probability confidence scores of each candidate direction set.

[0019] At least based on the total probability confidence of each candidate direction set, a score is calculated for each candidate direction set, and the score is positively correlated with the total probability confidence.

[0020] Select one of the top M candidate direction sets according to the scores from largest to smallest, and take the middle direction of the view range formed by the target candidate direction sets as the candidate direction selected this time, M≥1.

[0021] Preferably, calculating the score for each candidate direction set based at least on the total probability confidence of each candidate direction set includes:

[0022] The total probability confidence of each candidate direction set is used as the score for each candidate direction set.

[0023] Preferably, calculating the score for each candidate direction set based at least on the total probability confidence of each candidate direction set includes:

[0024] Determine the middle direction of the view range formed by each set of candidate directions, and calculate the turning cost of the robot turning to the middle direction of each set of candidate directions.

[0025] Based on the total probability confidence and turning cost of each candidate direction set, a score is calculated for each candidate direction set, and the score is negatively correlated with the turning cost.

[0026] Preferably, the process of calculating the probability distribution of user sound sources originating from multiple directions using a sound source localization method based on the microphone array includes:

[0027] The sound signal received by each microphone in the microphone array is converted to the frequency domain, and the phase difference between the sound signals received by different microphones at the same time is calculated;

[0028] The phase difference is input into the pre-trained sound source localization neural network model to obtain the probability distribution of user sound sources from various directions output by the model.

[0029] Preferably, the process of controlling the robot to turn to the candidate direction includes:

[0030] The robot is controlled to rotate toward the candidate direction at a first rotational angular velocity, and the rotational angular velocity is reduced when the robot rotates to a set angle away from the candidate direction.

[0031] Preferably, the setting of the steering cutoff condition includes any one or more of the following combinations:

[0032] The robot has reached a set threshold number of turns.

[0033] The robot's turning angle has reached the set angle threshold.

[0034] Preferably, it further includes:

[0035] If the target user is not identified after the set turning off conditions are met, a guiding voice prompt will be output.

[0036] Preferably, it further includes:

[0037] After identifying the target user, the robot determines the user's posture information based on the image of the target user collected by the image sensor, and determines the interaction intent by combining the target user's voice commands, and controls the robot to execute the task corresponding to the interaction intent.

[0038] Secondly, a robot interaction control device is provided, wherein the robot is equipped with a microphone array and an image sensor, and the interaction control device includes:

[0039] The sound source localization unit is used to calculate the probability distribution of the user's sound source coming from multiple directions based on the microphone array and the sound source localization method after receiving a voice command.

[0040] The steering control unit is used to select candidate directions in descending order of probability confidence according to the probability distribution, and after each selection of a candidate direction, control the robot to turn to the candidate direction and acquire the image collected by the image sensor.

[0041] The image recognition unit is used to determine whether a target user is identified based on the image. If the target user is identified, the selection of candidate directions is stopped. If the target user is not identified, the next candidate direction is selected until the target user is identified or the set turning cutoff condition is met.

[0042] Thirdly, a robot is provided, comprising: a robot body and a microphone array, an image sensor, a processor and a travel unit disposed on the robot body;

[0043] The microphone array is used to collect voice commands and send them to the processor;

[0044] The image sensor is used to acquire images and send them to the processor;

[0045] The traveling unit is used to perform steering or moving actions under the control of the processor;

[0046] The processor is used to execute the robot interaction control method described above.

[0047] Fourthly, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the robot interactive control method as described above.

[0048] Using the above technical solution, the robot interaction control method of this application, after receiving a voice command, calculates the probability distribution of the user's sound source from multiple directions based on the microphone array using a sound source localization method. This probability distribution represents the probability confidence level of the user's sound source in different directions. Based on this, candidate directions are selected sequentially according to the probability distribution and in descending order of probability confidence. After each selection of a candidate direction, the robot is controlled to turn towards the selected candidate direction and acquire an image collected by the image sensor on the robot. Based on the image, it is determined whether the target user has been identified. If the target user has been identified, the selection of candidate directions can be stopped. If the target user has not been identified, the next candidate direction can be selected until the target user has been identified or the set turning cutoff condition has been met. Obviously, the solution in this application uses the sound source localization method to calculate the probability distribution, and then selects different candidate directions in turn according to the probability confidence level. When the target user is not identified in a candidate direction, the next candidate direction is selected, which ultimately increases the probability of locating the target user direction, that is, increases the success rate of user interaction. In particular, when the robot is close to obstacles or there are other interfering sound sources in the environment, the solution in this application can greatly improve the success rate of user interaction. Attached Figure Description

[0049] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0050] Figure 1 A flowchart of a robot interaction control method provided in an embodiment of this application;

[0051] Figure 2 This example illustrates the process of dividing the plane in which the robot is located into candidate directions and selecting candidate directions.

[0052] Figure 3 This example illustrates another method for dividing the plane in which the robot is located into candidate directions and selecting candidate directions.

[0053] Figure 4 This is a schematic diagram of a robot interaction control device provided in an embodiment of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] This application provides a robot interaction control scheme that allows users to issue commands to the robot via voice. After receiving the voice command, the robot can locate the user's direction, identify the user's posture information through an image sensor, determine the user's interaction intention, and execute the corresponding task.

[0056] The robot interaction control solution of this application can be applied to various types of intelligent robots, commonly including home service robots such as cleaning robots, companion robots, and transport robots. The robot of this application is equipped with a microphone array and an image sensor. The microphone array can receive the user's voice and perform sound source localization. The image sensor can identify the presence of a target user in the environment and recognize the user's posture information. Furthermore, the robot also has a propulsion unit that can perform turning, moving, and other actions under the control of a processor.

[0057] Next, the robot interaction control method provided in the embodiments of this application will be described, which may specifically include the following steps:

[0058] Step S100: Upon receiving a voice command, the probability distribution of the user's voice source originating from multiple directions is calculated based on the microphone array using a sound source localization method.

[0059] Specifically, the robot can receive voice commands from users through a set microphone array. After receiving the voice command, the robot can calculate the probability distribution of the user's voice source from multiple directions based on the microphone array and a sound source localization method. That is, it can calculate the confidence level of the probability that the user's voice source is located in various set angle directions around the robot.

[0060] For example, the plane on which the robot is located can be divided into units of set angles, with the robot as the center. Then, the probability confidence level of the user's sound source in each direction after each division can be calculated. The higher the probability confidence level, the higher the likelihood that the user's sound source exists in the corresponding direction.

[0061] Step S110: According to the probability distribution, candidate directions are selected sequentially in descending order of probability confidence. After each candidate direction is selected, the robot is controlled to turn to the candidate direction and the image collected by the image sensor is acquired.

[0062] Specifically, the probability distribution obtained in the previous step indicates the confidence level of the probability of a user's sound source existing in different directions. In this step, candidate directions can be selected sequentially in descending order of this probability confidence level. For example, the first candidate direction with the highest probability confidence level can be selected first, and then the robot can be controlled to turn towards this first candidate direction, acquiring an image from the image sensor. The image acquired by the image sensor can be an image acquired after the robot turns towards the first candidate direction, or an image acquired by the image sensor during the robot's turn towards the first candidate direction.

[0063] Step S120: If the target user is identified based on the image, stop selecting candidate directions; otherwise, continue selecting the next candidate direction until the target user is identified or the set turning cutoff condition is met.

[0064] Specifically, after selecting a candidate direction each time, it can be determined whether the target user has been identified based on the image acquired by the image sensor, that is, whether the target user is contained in the image. If the target user is detected, it can be determined that the target user is in the currently selected candidate direction, that is, the direction of the user's sound source is determined. If the target user is not detected, it means that there is no target user in the currently selected candidate direction. The next candidate direction needs to be selected in descending order of probability confidence, and the process of acquiring the image acquired by the image sensor and identifying the target user in the image is repeated until the target user is identified after a candidate direction is selected, or the selection of the next candidate direction is stopped when the set turning cutoff condition is reached.

[0065] Optionally, the set steering cutoff conditions may include any one or more of the following combinations:

[0066] The number of times the robot turns reaches a set threshold, and the robot's turning angle reaches a set threshold.

[0067] For example, the threshold number of times can be set to 2 or other values. Taking 2 as an example, it means that the robot can select a maximum of 2 candidate directions and turn a maximum of 2 times. If the target user is not identified during the 2 turns, the robot stops selecting candidate directions, that is, stops the subsequent turning process to find the target user.

[0068] Setting an angle threshold can limit the robot's total turning coverage to the maximum angle value.

[0069] Furthermore, if the target user is not identified after the set turning deadline is reached, a guiding voice prompt can be output to guide the user to try a new round of interaction.

[0070] The robot interaction control method provided in this application, upon receiving a voice command, calculates the probability distribution of the user's sound source from multiple directions using a sound source localization method. This probability distribution represents the confidence level of the user's sound source in different directions. Based on this, candidate directions are selected sequentially according to the probability distribution, in descending order of probability confidence. After each selection of a candidate direction, the robot is controlled to turn towards that selected direction, and an image is acquired by the image sensor on the robot. Based on the image, it is determined whether the target user has been identified. If the target user is identified, the selection of candidate directions can be stopped; if not, the next candidate direction can be selected until the target user is identified or a set turning cutoff condition is met. Clearly, the robot in this application uses the probability distribution calculated by the sound source localization method and then sequentially selects different candidate directions according to the probability confidence level. When the target user is not identified in a candidate direction, the next candidate direction is selected, ultimately increasing the probability of locating the target user's direction, thus improving the success rate of user interaction. This is particularly beneficial when the robot is near obstacles or when there are other interfering sound sources in the environment; this solution can significantly improve the success rate of user interaction.

[0071] In some embodiments of this application, after identifying the target user according to the aforementioned method, the target user's posture information can be further determined based on the image of the target user collected by the image sensor, and the interaction intention can be determined by combining the target user's voice commands, and the robot can be controlled to perform the task corresponding to the interaction intention.

[0072] For example, after identifying the target user, an image sensor can be used to capture the user's image. The target user can then issue task commands via gestures, postures, and voice. For instance, the user points to an area on the ground and gives the voice command, "Clean this area." The robot, by capturing the user's image, can identify the target area pointed to by the user's gesture, designate that area as the work area, and perform the cleaning work on that area.

[0073] In some embodiments of this application, the process of calculating the probability distribution of user sound sources from multiple directions using the sound source localization method in step S100 above will be described.

[0074] In this embodiment, various sound source localization methods can be employed to calculate the probability distribution of user sound sources originating from multiple directions. For example, an algorithm based on relative time delay estimation can be used, such as estimating the time delay difference between signals in the microphone array using generalized cross-correlation (GCC), thereby determining the probability distribution of user sound sources originating from multiple directions.

[0075] Furthermore, this embodiment also provides a sound source localization method, which is specifically implemented using a pre-trained sound source localization neural network model.

[0076] When training the sound source localization neural network model, sample sound signals collected by the microphone array on the robot from test sound sources in various scenarios can be obtained in advance. The sample sound signals are then converted to the frequency domain, and the phase difference between the sample sound signals received by different microphones at the same time is calculated. The calculated phase difference is used as a training sample and input into the sound source localization neural network model. The model is used to predict the probability distribution of sound sources from various directions, and the true direction of the test sound source is used as the sample label. The sound source localization neural network model is trained using a set loss function.

[0077] When calculating the phase difference between sample sound signals received by different microphones at the same time, the phase difference can be calculated for each sample sound signal received by different microphones at each time. Alternatively, several key times can be selected according to a set strategy, and the phase difference can be calculated for each key time for the sample sound signals received by different microphones at each key time.

[0078] Once the trained sound source localization neural network model is obtained, it can be used to localize the sound source of the currently received speech command. Specifically:

[0079] Upon receiving a voice command, the system converts the sound signal received by each microphone in the microphone array to the frequency domain and calculates the phase difference between the sound signals received by different microphones at the same time.

[0080] The phase difference is input into the pre-trained sound source localization neural network model to obtain the probability distribution of user sound sources from various directions output by the model.

[0081] The sound source localization neural network model provided in this embodiment is used for sound source localization. Since this application can train the model using training data from various scenarios, the trained sound source localization neural network model is applicable to sound source localization tasks in a variety of different scenarios, such as complex environments (with obstacles) or scenarios with interfering sound sources. Therefore, when using the trained sound source localization neural network model for user sound source localization, the results are more accurate. Specifically, the calculated probability distribution of the user's sound source originating from multiple directions can be used to mark the true direction of the user's sound source with higher probability confidence. This facilitates rapid subsequent location of the target user's direction, improving the efficiency of human-computer interaction.

[0082] In some embodiments of this application, the process of selecting candidate directions in descending order of probability confidence according to the probability distribution in step S110 is described.

[0083] In one optional implementation, each time a candidate direction is selected, one of the top N (N≥1) candidate directions is selected from the candidate directions not yet covered by the viewpoint, sorted by probability confidence from highest to lowest. Specifically, the candidate directions not yet covered by the viewpoint are: within the plane where the robot is located, candidate directions excluding all candidate directions already covered by the horizontal viewpoint of the image sensor when the robot is in each of the selected candidate directions.

[0084] Where N can be 1 or any other positive integer greater than 1. When N is 1, the candidate direction with the highest probability confidence is selected according to the probability distribution. When N is a positive integer greater than 1, considering that the probability confidence of the top N candidate directions is relatively high, a candidate direction can be randomly selected from the top N candidate directions, or a candidate direction can be selected from them according to other strategies.

[0085] Taking an N value of 1 as an example, when initially selecting candidate directions, none of the candidate directions on the robot's plane are covered by the viewpoint. Therefore, the candidate direction with the highest probability confidence is selected and defined as the first candidate direction. If the target user is not identified in the first candidate direction, the second candidate direction is selected. Specifically, in the robot's plane, all candidate directions covered by the horizontal viewpoint of the image sensor when the robot is in the first candidate direction are excluded. Among the remaining candidate directions, the one with the highest probability confidence is selected as the second candidate direction. This process continues until the target user is identified or the set turning cutoff condition is met.

[0086] Reference Figure 2 As shown, assuming the robot's location is the center, the plane containing the robot is divided into 72 candidate directions, each at 5° intervals. Figure 2 Only five candidate directions are shown in the example; the rest are not shown. The probability distribution includes the probability confidence level for each candidate direction. The horizontal viewing angle of the image sensor is set to 25°.

[0087] Assumption Figure 2If the probability confidence level is highest in the d1 direction, then d1 is selected as the first candidate direction. Once the robot turns to the d1 direction, the horizontal field of view of the image sensor covers the d1 direction and two directions to the left and right of it, for a total coverage of 25°. If the target user is not detected in the image captured by the image sensor, the next candidate direction needs to be selected. In this case, the candidate direction with the highest probability confidence level is selected again from the remaining 340° of candidate directions as the second candidate direction, and so on, until the target user is detected or the set turning cutoff condition is met.

[0088] This embodiment also provides another method for selecting candidate directions, the specific process of which includes the following steps:

[0089] S11. Each time a candidate direction is selected, among the candidate directions that are not yet covered by the viewpoint, the horizontal viewpoint range of the image sensor is used as the window length, and a set step angle is used as the sliding step to obtain a set of candidate directions. The set step angle is smaller than the window length.

[0090] Reference Figure 3 As shown, if the plane where the robot is located is still divided into directions of 5° each, then the entire plane is divided into 72 candidate directions. Figure 2 Only a few candidate directions are shown in the example; the rest are not shown.

[0091] Taking the initial selection of candidate directions as an example, assuming a clockwise sliding window operation is performed from direction d1, the candidate direction set 1, consisting of all candidate directions covered by the first window, is: {d1,d2,d3,d4,d5}. Assuming the sliding step angle is 5°, the candidate direction set 2, consisting of all candidate directions covered by the second window, is: {d2,d3,d4,d5,d6}. Sliding windows sequentially downwards yields several candidate direction sets.

[0092] S12. Sum the probability confidence scores of each candidate direction in each candidate direction set to obtain the total probability confidence score of each candidate direction set.

[0093] Specifically, the above steps yield several candidate direction sets, each containing several candidate directions. The probability confidence scores of each candidate direction in each candidate direction set can be summed, and the result is used as the total probability confidence score of the corresponding candidate direction set. This score represents the probability that the target user can be identified in the acquired image when the robot turns so that the horizontal view of the image sensor covers each candidate direction in the candidate direction set.

[0094] S13. Calculate a score for each candidate direction set based at least on the total probability confidence level of each candidate direction set, wherein the score is positively correlated with the total probability confidence level.

[0095] In one alternative implementation, this step can directly use the total probability confidence of each candidate direction set as the score of each candidate direction set.

[0096] In another optional implementation, in addition to considering the probability confidence of candidate directions, the robot's turning cost can be further considered. By combining these two factors, the score of each candidate direction set is calculated, thereby balancing the robot's turning cost and the probability confidence of identifying the target user. Specifically, this may include the following steps:

[0097] S131. Determine the middle direction of the view range composed of each candidate direction set, and calculate the turning cost of the robot turning to the middle direction of each candidate direction set.

[0098] Specifically, in this embodiment, it is assumed that the image sensor is positioned directly in front of the robot. When the robot turns to the middle direction of the visual range composed of each set of candidate directions, the vision of the image sensor exactly covers the visual range composed of each set of candidate directions.

[0099] In this embodiment, the turning cost of the robot turning to the middle direction of each candidate direction set is further calculated. This turning cost can be measured by the turning angle, and the larger the turning angle, the higher the turning cost.

[0100] S132. Calculate the score for each candidate direction set based on the total probability confidence and turning cost of each candidate direction set.

[0101] The score is positively correlated with the total probability confidence level and negatively correlated with the turning cost.

[0102] S14. Select one of the top M (M≥1) candidate direction sets according to the scores from largest to smallest, and take the middle direction of the view range formed by the target candidate direction sets as the candidate direction selected this time.

[0103] Where M can be 1 or any other positive integer greater than 1. When M is 1, the target candidate direction set with the highest score is selected. When M is a positive integer greater than 1, considering that the scores of the top M candidate direction sets are all relatively large, a target candidate direction set can be randomly selected from the top M candidate direction sets, or a target candidate direction set can be selected from them according to other strategies. The middle direction of the visual range formed by the target candidate direction sets is taken as the candidate direction selected this time. When the robot turns to this candidate direction, the visual range of the image sensor coincides with the visual range formed by the target candidate direction sets, so the probability of detecting the target user in the acquired image is maximized.

[0104] In some embodiments of this application, the implementation process of determining whether a target user is identified based on an image in step S120 is described.

[0105] In one alternative implementation, without considering the identity of the user giving the voice command, an image detection method can be used to detect whether the image captured by the image sensor contains a human body. If so, the target user can be identified; otherwise, the target user cannot be identified.

[0106] In another optional implementation, considering that there may be multiple users in the robot's environment in real-world application scenarios, this embodiment provides a solution to avoid the robot misidentifying the user issuing the voice command:

[0107] First, the image sensor detects whether the image it captures contains a human body. If not, it can be determined that the target user has not been identified. If so, the identity of the human body in the image can be further determined to be consistent with the identity of the user who issued the voice command. If they are consistent, it can be determined that the target user has been identified. If they are inconsistent, it can be determined that the target user has not been identified.

[0108] Specifically, when determining the identity of a human being contained in an image, the human information identified in the image can be compared with that of a registered user. For example, the identified face can be compared with the face of a registered user to determine the identity of the human being identified in the image. Alternatively, the age, gender, and physical characteristics of the identified human body can be compared with the information of a registered user to determine the identity of the human being identified in the image.

[0109] For users who issue voice commands, their identity can be determined based on voiceprint recognition.

[0110] In the solution provided in this embodiment, by comparing the identity of the user in the image captured by the image sensor with the identity of the user who issued the voice command, the direction of the user who issued the voice command can be more accurately located, thus avoiding the problem of misidentification.

[0111] In some embodiments of this application, considering that if the angular velocity is too high when the robot turns in place, the image captured by the image sensor is prone to motion blur, interfering with the recognition accuracy of subsequent image recognition algorithms, this embodiment provides a solution that balances interaction efficiency and accuracy. During the process of controlling the robot to turn towards a candidate direction, the robot can be controlled to rotate towards the candidate direction at a first rotational angular velocity, and the rotational angular velocity can be reduced when it reaches a set angle from the candidate direction. That is, the robot can rotate towards the candidate direction in a manner that is initially faster and then slows down, ensuring that the image captured by the image sensor in the candidate direction does not exhibit motion blur, thus balancing interaction efficiency and image recognition accuracy.

[0112] The robot interaction control device provided in the embodiments of this application is described below. The robot interaction control device described below can be referred to in correspondence with the robot interaction control method described above.

[0113] See Figure 4 , Figure 4 This is a schematic diagram of the structure of a robot interaction control device disclosed in an embodiment of this application.

[0114] like Figure 4 As shown, the device may include:

[0115] The sound source localization unit 11 is used to calculate the probability distribution of the user's sound source coming from multiple directions based on the microphone array and the sound source localization method after receiving a voice command.

[0116] The steering control unit 12 is used to select candidate directions in descending order of probability confidence according to the probability distribution, and after each selection of a candidate direction, control the robot to turn to the candidate direction and acquire the image collected by the image sensor.

[0117] The image recognition unit 13 is used to determine whether a target user is recognized based on the image. If the target user is recognized, the selection of candidate directions is stopped. If the target user is not recognized, the next candidate direction is selected until the target user is recognized or the set turning cutoff condition is met.

[0118] Optionally, the process by which the image recognition unit determines whether a target user has been identified based on the image includes:

[0119] The system detects whether the image contains a human body. If it does, the target user is identified; otherwise, the target user is not identified.

[0120] or,

[0121] The system detects whether the image contains a human body. If not, it determines that the target user has not been identified. If so, it determines whether the identity of the human body in the image matches the identity of the user who issued the voice command. If they match, it determines that the target user has been identified. If they do not match, it determines that the target user has not been identified.

[0122] Optionally, the process by which the steering control unit selects candidate directions sequentially according to the probability distribution and in descending order of probability confidence includes:

[0123] Each time a candidate direction is selected, among the candidate directions that are not yet covered by the viewpoint, one of the top N candidate directions is selected according to the probability confidence level from largest to smallest. The candidate directions that are not yet covered by the viewpoint are: in the plane where the robot is located, the candidate directions that are excluded from all candidate directions covered by the horizontal viewpoint of the image sensor when the robot is in each of the selected candidate directions, N≥1.

[0124] In another scenario, the process by which the steering control unit selects candidate directions sequentially according to the probability distribution and in descending order of probability confidence includes:

[0125] Each time a candidate direction is selected, among the candidate directions that are not yet covered by the viewpoint, the horizontal viewpoint range of the image sensor is used as the window length, and a set step angle is used as the sliding step to obtain a set of candidate directions. The set step angle is smaller than the window length.

[0126] The total probability confidence of each candidate direction set is obtained by summing the probability confidence scores of each candidate direction set.

[0127] At least based on the total probability confidence of each candidate direction set, a score is calculated for each candidate direction set, and the score is positively correlated with the total probability confidence.

[0128] Select one of the top M candidate direction sets according to the scores from largest to smallest, and take the middle direction of the view range formed by the target candidate direction sets as the candidate direction selected this time, M≥1.

[0129] Optionally, the process by which the steering control unit calculates a score for each candidate direction set based at least on the total probability confidence of each candidate direction set includes:

[0130] The total probability confidence of each candidate direction set is used as the score for each candidate direction set.

[0131] Alternatively, the process by which the steering control unit calculates a score for each candidate direction set based at least on the total probability confidence of each candidate direction set includes:

[0132] Determine the middle direction of the view range formed by each set of candidate directions, and calculate the turning cost of the robot turning to the middle direction of each set of candidate directions.

[0133] Based on the total probability confidence and turning cost of each candidate direction set, a score is calculated for each candidate direction set, and the score is negatively correlated with the turning cost.

[0134] Optionally, the process by which the sound source localization unit calculates the probability distribution of user sound sources originating from multiple directions based on the microphone array using a sound source localization method includes:

[0135] The sound signal received by each microphone in the microphone array is converted to the frequency domain, and the phase difference between the sound signals received by different microphones at the same time is calculated;

[0136] The phase difference is input into the pre-trained sound source localization neural network model to obtain the probability distribution of user sound sources from various directions output by the model.

[0137] Optionally, the process by which the steering control unit controls the robot to turn toward the candidate direction includes:

[0138] The robot is controlled to rotate toward the candidate direction at a first rotational angular velocity, and the rotational angular velocity is reduced when the robot rotates to a set angle away from the candidate direction.

[0139] Optionally, the device of this application may further include: a prompt voice output unit, used to output a guiding prompt voice when the target user is not identified after the set turning off conditions are met.

[0140] Optionally, the apparatus of this application may further include:

[0141] The interaction intent recognition unit is used to determine the posture information of the target user based on the image of the target user collected by the image sensor after the target user is identified, and to determine the interaction intent by combining the voice command of the target user, and to control the robot to execute the task corresponding to the interaction intent.

[0142] This application also provides a robot, including a robot body and a microphone array, image sensor, processor and travel unit disposed on the robot body.

[0143] The microphone array is used to collect voice commands and send them to the processor;

[0144] The image sensor is used to acquire images and send them to the processor;

[0145] The traveling unit is used to perform steering or moving actions under the control of the processor;

[0146] The processor is used to execute the robot interaction control method described in the above embodiments.

[0147] The robot provided in this application can be a home service robot, such as a robot vacuum cleaner or a delivery robot, or a factory robot, such as a handling robot.

[0148] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used to execute various steps of the robot interaction control method described in the above embodiments.

[0149] Optionally, the refined and extended functions of the program can be found in the description above.

[0150] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0151] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0152] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robot interactive control method, characterized in that, The robot is provided with a microphone array and an image sensor, and the interactive control method comprises: After receiving the voice instruction, the probability distribution of the user sound source from multiple directions is calculated based on the microphone array through a sound source positioning method; According to the probability distribution, the candidate directions are sequentially selected in the order of probability confidence from large to small, and after each candidate direction is selected, the robot is controlled to turn to the candidate direction, and the image collected by the image sensor is obtained; wherein, according to the probability distribution, the candidate directions are sequentially selected in the order of probability confidence from large to small, which comprises: when each candidate direction is selected, the horizontal visual angle range of the image sensor is taken as the window length, and the set step angle is taken as the sliding step to slide the window in each candidate direction which has not been covered by the visual angle, so as to obtain a plurality of candidate direction sets, and the set step angle is smaller than the window length; the probability confidence of each candidate direction in each candidate direction set is summed to obtain the total probability confidence of each candidate direction set; the middle direction of the visual angle range composed of each candidate direction set is determined, and the turning cost of the robot turning to the middle direction of each candidate direction set is calculated, the score of each candidate direction set is calculated based on the total probability confidence and the turning cost of each candidate direction set, the score is in a positive correlation with the total probability confidence and in a negative correlation with the turning cost, and the total probability confidence represents the probability of identifying the target user in the collected image when the horizontal visual angle of the image sensor covers each candidate direction in the candidate direction set after the robot turns; one target candidate direction set in the topM candidate direction sets is selected in the order of score from large to small, and the middle direction of the visual angle range composed of the target candidate direction set is taken as the candidate direction selected this time, and M≥1; It is determined whether the target user is identified based on the image, if the target user is identified, the selection of the candidate direction is stopped, if the target user is not identified, the next candidate direction is continuously selected until the target user is identified or the set turning cutoff condition is reached.

2. The method of claim 1, wherein, It is determined whether the target user is identified based on the image, which comprises: If the image contains a human body, it is determined that the target user is identified, otherwise, it is determined that the target user is not identified; Or, If the image does not contain a human body, it is determined that the target user is not identified, if the image contains a human body, it is determined whether the identity of the human body in the image is consistent with the identity of the user who issues the voice instruction, if yes, it is determined that the target user is identified, if no, it is determined that the target user is not identified.

3. The method of claim 1, wherein, The process of calculating the probability distribution of the user sound source from multiple directions based on the microphone array through the sound source positioning method comprises: The sound signals received by each microphone in the microphone array are converted to the frequency domain, and the phase difference between the sound signals received by different microphones at the same time is calculated; The phase difference is input into a pre-trained sound source positioning neural network model to obtain the probability distribution of the user sound source from each direction output by the model.

4. The method of claim 1, wherein, The process of controlling the robot to turn to the candidate direction comprises: controlling the robot to turn to the candidate direction at a first rotation angular velocity, and reducing the rotation angular velocity when turning to a set angle from the candidate direction.

5. The method of claim 1, wherein, The set turning-off condition comprises any one or a combination of the following: the number of times of robot turning reaches a set number threshold; the angle of robot turning reaches a set angle threshold.

6. The method of claim 1, wherein, Further comprising: if the target user is not identified after the set turning-off condition is reached, outputting a guiding prompt voice.

7. The method according to any one of claims 1 to 6, characterized in that, Further comprising: after the target user is identified, determining the posture information of the target user based on the image of the target user collected by the image sensor, combining the voice instruction of the target user to determine the interaction intention, and controlling the robot to perform a task corresponding to the interaction intention.

8. A robot interaction control device, characterized by, The robot is provided with a microphone array and an image sensor, and the interaction control device comprises: a sound source positioning unit, configured to, after receiving the voice instruction, calculate the probability distribution of the user sound source coming from multiple directions based on the microphone array through a sound source positioning method; a turning control unit, configured to, according to the probability distribution, sequentially select candidate directions in the order of probability confidence from large to small, and after each candidate direction is selected, control the robot to turn to the candidate direction and obtain the image collected by the image sensor; an image recognition unit, configured to determine whether the target user is identified based on the image, if the target user is identified, stop selecting the candidate direction, and if the target user is not identified, continue to select the next candidate direction until the target user is identified or the set turning-off condition is reached; wherein the turning control unit sequentially selects the candidate directions according to the probability distribution in the order of probability confidence from large to small, comprising: when each candidate direction is selected, in each candidate direction that has not been covered by the visual angle, taking the horizontal visual angle range of the image sensor as the window length and taking the set step angle as the sliding step to perform sliding window to obtain a plurality of candidate direction sets, and the set step angle is smaller than the window length; summing the probability confidence of each candidate direction in each candidate direction set to obtain the total probability confidence of each candidate direction set; determining the middle direction of the visual angle range formed by each candidate direction set, and calculating the turning cost of the robot turning to the middle direction of each candidate direction set, calculating the score of each candidate direction set based on the total probability confidence and the turning cost of each candidate direction set, the score is positively correlated with the total probability confidence and negatively correlated with the turning cost, the total probability confidence represents the probability of being able to identify the target user in the collected image when the horizontal visual angle of the image sensor covers each candidate direction in the candidate direction set after the robot turns; selecting one target candidate direction set from the topM candidate direction sets in the order of score from large to small, and taking the middle direction of the visual angle range formed by the target candidate direction set as the candidate direction selected this time, and M≥1.

9. A robot, characterized in that comprising: A robot body and a microphone array, an image sensor, a processor and a traveling unit arranged on the robot body; The microphone array is configured to collect voice instructions and send the voice instructions to the processor; The image sensor is configured to collect images and send the images to the processor; The traveling unit is configured to perform a turning or moving action under the control of the processor; The processor is configured to perform the robot interaction control method according to any one of claims 1-7.

10. A storage medium having stored thereon a computer program, characterized in that The computer program, when executed by a processor, implements the robot interaction control method according to any one of claims 1-7.

Images