Robot interaction method, robot, apparatus, and storage medium
By using vision sensors on the robot for abnormal detection and command recognition, the problem of low human-computer interaction efficiency in staff hands or noisy environment is solved, and more efficient and reliable interactive processing is achieved.
Patent Information
- Application Number
- PCT/CN2023/139866
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-16
- Filing Date
- 2023-12-19
- Publication Date
- 2024-11-21
AI Technical Summary
In the case where the staff’s hands are occupied or the environment is noisy, human-computer interaction is problematic with low efficiency, especially when the robot needs to perform tasks or move to a target point, touch screens and voice interaction methods in the prior art are inefficient.
By installing a vision sensor on the robot, using it to perform abnormal detection during movement, and pause detection when reaching the interaction point, identifying human-computer interaction instructions, realizing interactive processing based on visual sensor data, and avoiding touch screen and voice interaction Inefficient efficiency and high error rate.
Without increasing hardware costs, the efficiency and reliability of human-computer interaction are improved, and the problems of low interaction efficiency when using hands and high error rates in noisy environments are solved.
Smart Images

Figure CN2023139866_21112024_PF_FP_ABST
Abstract
Description
Robot interaction method, robot, device and storage medium
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 16, 2023, with application number 202310545904.1 and invention name “A robot interaction method, robot, device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of robotics technology, and in particular to a robot interaction method, a robot, a device, and a storage medium. Background Art
[0003] With the rapid development of artificial intelligence (AI), robotics has made significant progress in recent years, with robots being widely used in a growing number of scenarios. For example, robots are used in industrial material delivery scenarios, and service robots are used in restaurants, hotels, and guesthouses.
[0004] When a robot needs to start a mission or reach a destination, it often needs to interact with a human operator to obtain instructions for the task to be executed. For example, after placing a meal on the robot's tray, a human operator needs to initiate a meal delivery instruction.
[0005] In existing technologies, human-machine interaction typically involves workers issuing commands by touching the robot's screen or interacting with the robot via voice. However, in real-world scenarios, workers often find their hands occupied, making it difficult to operate the screen, resulting in low touch-based interaction efficiency. Furthermore, robots often operate in noisy environments, which in turn reduces voice recognition efficiency. Technical issues
[0006] The present application provides a robot interaction method, robot, device and storage medium to solve the problem that the staff's hands are occupied and it is inconvenient to operate the screen, or the human-computer interaction efficiency is low in a noisy working environment. Technical Solutions
[0007] According to one aspect of the present application, a robot interaction method is provided, which is applied to a processor of a robot, wherein the robot is provided with a visual sensor, and the method comprises:
[0008] During the movement of the robot toward the human-machine interaction point, abnormality detection is performed through the data collected by the visual sensor;
[0009] When the robot reaches the human-machine interaction point, suspending the abnormality detection;
[0010] Human-computer interaction instructions are identified based on the collected data of the visual sensor.
[0011] According to another aspect of the present application, a robot is provided, comprising: a visual sensor and a processor; the processor and the visual sensor are communicatively connected;
[0012] The visual sensor is used to send collected data to the processor when the robot moves towards the human-machine interaction point and when it reaches the human-machine interaction point;
[0013] The processor is used to execute the robot interaction method described in any embodiment of the present application.
[0014] According to another aspect of the present application, a robot interaction device is provided, comprising:
[0015] The anomaly detection module is used to detect anomalies using data collected by the visual sensor during the robot's movement toward the human-machine interaction point;
[0016] An abnormality detection pause module, used to pause the abnormality detection when the robot reaches the human-machine interaction point;
[0017] The instruction recognition module is used to recognize human-computer interaction instructions based on the data collected by the visual sensor.
[0018] According to another aspect of the present application, an electronic device is provided, comprising:
[0019] at least one processor; and
[0020] a memory communicatively connected to the at least one processor; wherein,
[0021] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the robot interaction method described in any embodiment of the present application.
[0022] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the robot interaction method described in any embodiment of the present application when executed.
[0023] The technical solution of the embodiment of the present application uses the data collected by the visual sensor to detect anomalies during the robot's movement toward the human-machine interaction point. When the robot reaches the human-machine interaction point, the anomaly detection is suspended and the human-machine interaction command is identified based on the data collected by the visual sensor. By processing the data collected by the visual sensor differently in different working states of the robot, human-machine interaction is achieved, avoiding the shortcomings of touch screen interaction, such as low interaction efficiency when both hands are occupied, and high error rate of voice interaction in noisy environments. This solution can improve the efficiency and reliability of human-machine interaction without increasing hardware costs.
[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0026] FIG1 is a flow chart of a robot interaction method provided according to Embodiment 1 of the present application;
[0027] FIG2a is a flow chart of a robot interaction method provided according to Embodiment 2 of the present application;
[0028] FIG2b is a flowchart of human-computer interaction instruction recognition according to the second embodiment of the present application;
[0029] FIG3 is a flow chart of a robot interaction method provided according to Embodiment 3 of the present application;
[0030] FIG4 is a schematic structural diagram of a robot provided according to a fourth embodiment of the present application;
[0031] FIG5 is a schematic structural diagram of a robot interaction device provided according to a fifth embodiment of the present application;
[0032] FIG6 is a schematic diagram of the structure of an electronic device for implementing the robot interaction method according to an embodiment of the present application. Modes for Carrying Out the Invention
[0033] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.
[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0035] Example 1
[0036] FIG1 is a flowchart of a robot interaction method provided in accordance with a first embodiment of the present application. This embodiment can be applied to identifying human-machine interaction instructions based on data collected by the robot's visual sensor. This method can be applied to the robot's processor. It should be noted that the robot also needs to be provided with a visual sensor, such as a stereoscopic visual sensor. The visual sensor is used to detect environmental information of the collection area, so that the processor can identify the robot's environmental information based on the information collected by the visual sensor. In addition, the processor can also identify human-machine interaction instructions based on the information collected by the visual sensor.
[0037] The robot interaction method involved in the embodiments of the present application can be performed by a robot interaction device, which can be implemented in the form of hardware and / or software. The robot interaction device can be configured in various general-purpose computing devices, for example, the general-purpose computing device is a processor built into the robot. As shown in Figure 1, the method includes:
[0038] S110. During the movement of the robot toward the human-machine interaction point, abnormality detection is performed through data collected by the visual sensor.
[0039] Human-robot interaction points are a general term for the locations where robots and humans interact. These can be any location where a robot needs to receive work instructions from a human. For example, in a robot delivery scenario, these could be the robot's origin, the delivery starting point, or the delivery destination.
[0040] In an embodiment of the present application, the robot body is equipped with a visual sensor for detecting the environment in which the robot is located. For example, the robot is equipped with a stereo vision sensor. After the robot receives a call message from the user or task information sent by the server, during the movement toward the human-computer interaction point, the visual sensor located on the robot body collects information about the environment in which it is located during the movement in real time, and sends the collected information to the robot's processor. Based on the information collected by the visual sensor, the processor can perform anomaly detection during the robot's movement, such as obstacle detection and cliff detection. Specifically, the processor converts the collected data of the visual sensor into three-dimensional point cloud data, and determines whether there are obstacles or ground cliffs around the robot's location based on the three-dimensional point cloud data.
[0041] In a specific example, a food delivery robot responds to a call by moving to the food delivery port and opening the storage compartment. During this movement, the robot's stereo vision sensors collect real-time environmental information along its route and transmit it to a processor. The processor processes this data to determine whether there are static or dynamic obstacles along the route, as well as any cliffs on the road, to prevent collisions or falls down stairs.
[0042] S120: When the robot reaches the human-machine interaction point, suspend abnormality detection.
[0043] In an embodiment of the present application, after the robot reaches the human-machine interaction point, it needs to wait for the control command of the human to perform the next stage of the task. At this time, the robot suspends abnormality detection to avoid misidentification, such as avoiding misidentification of service personnel as obstacles, triggering error obstacle avoidance. The robot can switch from abnormality detection mode to human-machine interaction command recognition mode after detecting that the interactive command recognition conditions are met. The interactive command recognition conditions can be a command recognition request issued by the server after the robot stops moving, or it can be a pre-set command recognition condition detected by the robot's own sensors.
[0044] It is worth noting that in the human-computer interaction instruction recognition mode, the visual sensor does not stop working, but continues to send the collected data to the processor for processing. In the abnormality detection mode and the human-computer interaction instruction recognition mode, the processor processes the collected information of the visual sensor differently. Optionally, if the stereo vision sensor has an adjustable collection angle, when switching between the abnormality detection mode and the human-computer interaction instruction recognition mode, the collection angle of the stereo vision sensor can be adjusted according to the mode switched to, thereby improving accuracy and efficiency. For example, when switching from the abnormality detection mode to the human-computer interaction instruction recognition mode, the collection angle of the stereo vision sensor can be adjusted downward to detect the ground area near the robot to accurately identify the human-computer interaction instructions.
[0045] S130: Identify human-computer interaction instructions based on data collected by the visual sensor.
[0046] In an embodiment of the present application, after the working mode of the robot is converted from the abnormality detection mode to the human-computer interaction instruction recognition mode, the processor will identify the human-computer interaction instruction based on the collected data of the visual sensor. Specifically, the collected data of the visual sensor is converted into three-dimensional point cloud data. Furthermore, for the convenience of calculation, the three-dimensional point cloud data can be converted into the robot coordinate system. Finally, the collected data converted into the robot coordinate system is identified to determine the human-computer interaction instruction. For example, the converted collected data is compared with the pre-set instruction feature. When the two match, it is determined that the human-computer interaction instruction is identified. Among them, the human-computer interaction instruction can be a command to control the robot to close the hatch. The robot coordinate system is a right-handed coordinate system with the robot's optional center as the coordinate origin, the front of the robot as the positive direction of the X-axis, and the height direction of the robot as the positive direction of the Z-axis.
[0047] In a specific example, the food delivery robot responds to a call from a person, moves to the food delivery port, stops, and opens the storage compartment door. A tray capable of detecting items is pre-installed in the compartment. The tray detects when a waiter places a dish. If no other dishes are placed for a certain period of time (e.g., 20 seconds), the robot enters a human-machine interaction command recognition mode. The robot identifies the human-machine interaction command by identifying the data collected by the visual sensor. The human-machine interaction command can be a command to close the compartment door, or a command to close the compartment door and move to the delivery destination.
[0048] In another specific example, after the food delivery robot reaches the delivery destination, it stops and opens the storage compartment door. If the tray in the compartment detects that a waiter has removed a dish and that no other items have been moved for 20 seconds after the dish is removed, it activates the human-machine interaction command recognition mode and identifies the human-machine interaction command by identifying the data collected by the visual sensor. The human-machine interaction command can be a command to close the compartment door or a command to close the compartment door and move to the next delivery destination.
[0049] The technical solution of the embodiment of the present application uses the data collected by the visual sensor to detect anomalies during the robot's movement toward the human-machine interaction point. When the robot reaches the human-machine interaction point, the anomaly detection is suspended and the human-machine interaction command is identified based on the data collected by the visual sensor. By processing the data collected by the visual sensor differently in different working states of the robot, human-machine interaction is achieved, avoiding the shortcomings of touch screen interaction, such as low interaction efficiency when both hands are occupied, and high error rate of voice interaction in noisy environments. This solution can improve the efficiency and reliability of human-machine interaction without increasing hardware costs.
[0050] Example 2
[0051] Figure 2a is a flow chart of a robot interaction method provided in Example 2 of this application. This embodiment further refines the above embodiment and provides specific steps for identifying human-machine interaction instructions based on data collected by the visual sensor. As shown in Figure 2a, the method includes:
[0052] S210: When the robot moves toward the human-machine interaction point, abnormality detection is performed through the data collected by the visual sensor.
[0053] S220: When the robot reaches the human-machine interaction point, suspend abnormality detection.
[0054] S230: Control the projection light to display an interaction indicator in a preset instruction recognition area, so as to instruct the user to initiate an interaction action in the corresponding area based on the interaction indicator.
[0055] The projector is a device installed on the robot body that projects an interactive indicator on a designated area. The interactive indicator is used to indicate the location where the user should initiate an interaction. For example, if the projector projects an interactive indicator on the ground in front of the robot, it indicates that the user needs to initiate an interaction action in the corresponding area above the ground in front of the robot.
[0056] In an embodiment of the present application, when the robot reaches the human-computer interaction point and detects that the interaction instruction recognition conditions are met, it first controls the projection light to display the interaction indication mark in the preset instruction recognition area to instruct the user to initiate an interaction action in the interaction area associated with the instruction recognition area.
[0057] Optionally, while controlling the projection light to instruct the user to initiate an interactive action, the robot can also issue voice prompts to guide the user to initiate an interactive action in multiple dimensions to ensure that the user can receive the signal that the robot has started the human-computer interaction mode.
[0058] Exemplarily, the projector projects an interactive indicator on the command recognition area on the ground in front of the robot, which is used to instruct the user to perform a foot interaction action on the ground in front of the robot. Another exemplary embodiment is that the projector projects an interactive indicator on the wall or other plane in front of the robot, which is used to instruct the user to perform a specific interactive action in front of the robot. The interactive indicator can be a text-based prompt or an icon-based prompt, which is not specifically limited here. For example, the interactive instruction indicator can be a dot, an arrow, or a graphic of a specific human body part, such as a hand-shaped indicator or a foot-shaped indicator.
[0059] Optionally, the visual sensor is tilted downward, the installation position of the visual sensor is 0.3-2.0 meters from the ground, the installation position of the projection lamp is 0.1-1.0 meters from the ground, and the installation position of the visual sensor is higher than the installation position of the projection lamp.
[0060] This optional embodiment provides for the installation of the vision sensor and projection light: the vision sensor is tilted downward and installed in front of the robot, 0.3-2.0 meters from the ground. For example, if the robot is 1.3 meters tall, the vision sensor should be installed in front of the robot, 1.1-1.2 meters from the ground.
[0061] The projector is used to project the interactive indicator, which prompts the user to initiate foot interaction commands. Therefore, the interactive indicator is generally projected on the side of the robot where it picks up and places items, such as the ground in front of the robot. Given these projection requirements, the projector can be placed in front of the robot, 0.1-1.0 meters above the ground. To ensure that the interactive indicator projected by the projector is projected smoothly to the ground and is not blocked by waiters' actions to pick up food or the opening and closing of storage compartment doors, the projector can be placed at a lower height, for example, 0.2-0.3 meters from the ground.
[0062] The stereo vision sensor is mounted higher than the projector lamp, which expands its field of view and facilitates anomaly detection during robot motion. The projector lamp is mounted lower to minimize the impact of robot operation and other obstacles on the projection.
[0063] S240: Control the visual sensor to collect data associated with the interactive action.
[0064] In an embodiment of the present application, when the robot turns on the human-computer interaction instruction recognition mode, the projection lamp is controlled to project the interaction indication mark, and the visual sensor is controlled to collect sensor data of the instruction recognition area. The collected data at this time is used to identify the human-computer interaction instruction.
[0065] S250: Identify the collected data and obtain human-computer interaction instructions.
[0066] In an embodiment of the present application, after acquiring data collected by a visual sensor, the processor identifies the collected data and obtains human-computer interaction instructions. Specifically, the collected data is first converted from sensor data into point cloud data, and then the point cloud data is converted into a robot coordinate system. The processor can extract data features in the robot coordinate system and compare the data features with pre-set interaction action features to obtain human-computer interaction instructions. The processor can also filter the data in the robot coordinate system before extracting the data features to filter out interference points outside the instruction recognition area.
[0067] Optionally, the collected data is identified to obtain human-computer interaction instructions, including:
[0068] Convert the collected data into point cloud data, and then convert the point cloud data into the robot coordinate system to obtain the initial point set;
[0069] Screen the initial point set to obtain the interactive action points contained in the initial point set;
[0070] By identifying the interaction action points, the human-computer interaction instructions are determined.
[0071] In this optional embodiment, a specific method for identifying the collected data and obtaining human-computer interaction instructions is provided: first, the collected data is converted into three-dimensional point cloud data, and then the coordinates of the point cloud data are converted into the robot coordinate system to obtain an initial point set. Furthermore, since the initial point set contains all points within the detection range of the visual sensor, these points may contain other obstacle points and ground points outside the instruction recognition area in addition to the interactive action points. Therefore, the initial point set can be screened to filter out the ground points and other obstacle points contained therein to obtain the interactive action points. Finally, the human-computer interaction instructions are determined by identifying the interactive action points. For example, the horizontal size, vertical size and average height of the human-computer interaction point set are compared with a pre-set size range. If they are all within the set size range, it is determined that the human-computer interaction instruction has been detected.
[0072] Optionally, the initial point set is filtered to obtain the interactive action points contained in the initial point set, including:
[0073] In the initial point set, a transition point set located in the instruction recognition area is selected;
[0074] The ground points are filtered out at the transition points to obtain the interactive action points.
[0075] In this optional embodiment, a specific method is provided for filtering the initial point set to obtain the interactive action points contained in the initial point set: first, in the initial point set, the points located in the instruction recognition area are filtered out as the transition point set, wherein the instruction recognition area can be a rectangular area with a certain length and width in the robot coordinate system. Exemplarily, the instruction recognition area can be determined based on the overlapping area of the acquisition range of the visual sensor and the projection range of the projector. Furthermore, the transition point set can be filtered to filter out the ground points contained therein, and the remaining points can be used as interactive action points, wherein the ground points are points contained in the horizontal ground / non-horizontal ground in the area where the robot is located. Before performing interactive instruction recognition, the initial point set is filtered to avoid the influence of ground points and points outside the instruction recognition area on the interactive instruction recognition, thereby improving the recognition accuracy of interactive instructions.
[0076] Optionally, ground points are filtered out from the transition point set to obtain interactive action points, including at least one of the following:
[0077] According to the pre-set horizontal ground height range, the ground points are filtered out at the transition point to obtain the interactive action points;
[0078] The extracted ground parameters of the non-horizontal ground are converted to the robot coordinate system, and the ground points in the transition point concentration are filtered out according to the converted ground parameters to obtain the interactive action points.
[0079] In this optional embodiment, a specific method is provided for filtering out ground points at transition points to obtain interactive action points: when the ground at the robot's location is horizontal, the ground points are filtered out at transition points according to a pre-set horizontal ground height range to obtain interactive action points. Exemplarily, points with a Z-axis coordinate of 0 are filtered out, and the remaining points are obtained as interactive action points. As another example, considering that the visual sensor inevitably has measurement errors, resulting in the measured ground points being unable to be accurately distributed at a Z-axis coordinate of 0, it can be further extended to filter out points with a Z-axis coordinate within the range of [-h, h] to obtain the remaining points as interactive action points. h can be flexibly set according to the measurement accuracy of the visual sensor. For example, h can be 0.01-0.03 meters. By setting the h value, the accuracy of ground filtering can be improved.
[0080] When the ground where the robot is located is a slope, steps or uneven non-horizontal ground, the visual sensor can be used to first extract the ground point where the robot is currently located, and then the ground point can be converted to the robot coordinate system. Furthermore, the converted ground points are concentratedly filtered out at the transition point to obtain the interactive action point. Exemplarily, whether the ground where the robot is currently located is a non-horizontal ground can be determined based on the robot's sensors, such as whether the robot is currently tilted. It can also be determined based on the robot's current positioning information and the marked area of the non-horizontal ground on the map. For non-horizontal ground, the ground plane can be extracted in the robot's projection area to obtain the conversion relationship between the ground plane and the camera coordinate system. Based on this, the new ground can be filtered out to obtain the remaining obstacle point cloud, and the subsequent judgment steps can be continued.
[0081] Optionally, identifying the interaction action point to determine the human-computer interaction instruction includes:
[0082] Clustering the interaction action points according to the distances between the interaction action points to obtain at least one cluster;
[0083] In at least one cluster, determining a target cluster containing the most interactive action points, and determining a ratio of the number of target interactive action points in the target cluster to the number of interactive action points;
[0084] When the number ratio is greater than the set ratio threshold, determine whether the target cluster is consistent with the preset interaction characteristics based on the axial size, lateral size and axial normal vector distribution of the target cluster;
[0085] When the target cluster matches the preset interaction characteristics, the human-computer interaction instruction is determined based on the target cluster.
[0086] In this optional embodiment, a specific method for determining human-computer interaction instructions by identifying interactive action points is provided: first, the interactive action points are clustered according to the distance between the interactive action points to obtain at least one cluster. Further, the number of interactive action points contained in each cluster is obtained, the cluster containing the largest number of interactive action points is used as the target cluster, and the proportion of the number of target interactive action points contained in the target cluster among all interactive action points is calculated. The number proportion is compared with a set proportion threshold (for example, 80%). If the number proportion is less than or equal to the proportion threshold, it indicates that the obstacle information identified in the instruction recognition area is too messy and does not match the human-computer interaction situation, and it is considered that the human-computer interaction instruction has not been identified; if the number proportion is greater than the proportion threshold, it indicates that the obstacles are concentrated in one cluster, and there may be a human-computer interaction instruction, which requires further judgment.
[0087] Specifically, when the number of clusters exceeds a set threshold, the axial size, lateral size, average point height, and axial normal vector distribution of the target cluster are obtained and compared with pre-set interaction features. If the target cluster matches the preset interaction features, the human-computer interaction command is determined based on the target cluster. By comparing the multi-dimensional data of the template cluster with the preset interaction features, it is possible to avoid misidentification of interaction commands caused by the presence of other obstacles and prevent the target user's non-interaction command actions from being mistakenly identified as interaction command actions, thereby improving the reliability of interaction command recognition.
[0088] Preset interaction features can be pre-extracted based on interaction actions, including axial dimensions, lateral dimensions, and average point height. For example, if the interaction action involves a user's foot blocking the command recognition area, the user's foot can be abstracted as a cube to obtain the preset interaction features. In delivery scenarios like restaurants, users' hands are often occupied for picking up and placing items. Therefore, detecting foot interaction features can improve the robot's efficiency and reduce the burden on the user.
[0089] In addition, in order to further avoid the misrecognition of interaction commands due to the presence of other obstacles, the sensor data of the position within a set time period before the robot detects the human-computer interaction command can be backtracked.
[0090] For example, if a human-machine interaction command includes a door-opening command, if the robot detects a door-opening command, the processor will backtrack the stereoscopic vision sensor data at that location within a set time period before the door-opening command was detected. If no fixed obstacles appear in the command recognition area within the set time period, it is assumed that there are no fixed obstacles, and the corresponding operation can continue according to the recognized human-machine interaction command. If a fixed obstacle appears in the command recognition area within the set time period, it is assumed that an obstacle exists in that area, and the door-opening command will not be executed, ensuring the safety of the robot during operation. After recognizing the door-opening command, the user can also be prompted on the display screen to check for obstacles and move before retesting. After moving or rotating to an obstacle-free area, the robot can re-enable the human-machine interaction command detection mode to recognize human-machine interaction commands.
[0091] Optionally, based on the target cluster, determine the human-computer interaction instructions, including:
[0092] When the target cluster matches the preset interaction feature, the timing duration of the interaction detection timer is obtained; the interaction detection timer is used to time the duration of the identified interaction action;
[0093] When the timing reaches the set time threshold, the human-computer interaction instruction is determined based on the target cluster.
[0094] In this optional embodiment, a specific method for determining human-computer interaction instructions based on the target cluster is provided: when the target cluster matches the preset interaction feature, the timing duration of the interaction detection timer is obtained, wherein the interaction detection timer is used to time the duration of the identified interaction action. If the duration of the detected interaction action reaches the set time threshold, the human-computer interaction instruction can be determined based on the target cluster. By timing the duration of the detected interaction action, the human-computer interaction instruction is considered to be identified only if the interaction duration continues to exceed the set time threshold, which can avoid the misidentification of human-computer interaction instructions caused by short-lived obstacles or transient anomalies in sensor data.
[0095] In addition, a timeout timer can be started before the human-computer interaction command is recognized. If no interaction action is detected after the timeout timer exceeds the timeout threshold, it is considered that the human-computer interaction command has not been recognized within the effective waiting time. At this time, the projector light can be turned off, ending the human-computer interaction command recognition process.
[0096] In a specific example, as shown in FIG2b , when the robot reaches the human-computer interaction point and meets the interaction instruction detection conditions, the human-computer interaction instruction detection mode is turned on. First, the interaction detection timer and the timeout timer are initialized. Then, the projection lamp is controlled to project the interaction indication mark in the instruction recognition area, and the timing length of the timeout timer is continuously detected to see if it exceeds the timeout threshold. If the timeout threshold is exceeded, a prompt message indicating that the recognition timeout has occurred is output, the projection lamp is turned off, and the human-computer interaction instruction recognition process ends; if the timeout threshold is not exceeded, the interaction action is continuously recognized. If no interaction action is recognized, the timing length of the interaction detection timer is set to 0; if an interaction action is recognized, the timing length of the interaction detection timer is obtained, and it is determined whether the timing length exceeds the time threshold. If the time threshold is exceeded, a prompt message indicating that the recognition is successful is output, the corresponding interaction instruction is executed, the projection lamp is turned off, and the human-computer interaction instruction recognition process ends.
[0097] In addition, the progress bar of the interaction detection timer and the timeout timer can be displayed on the robot's display screen to facilitate the user to understand the recognition progress of the human-computer interaction command. After exiting the human-computer interaction command recognition mode, it switches back to the anomaly detection mode.
[0098] The technical solution of the embodiment of this application controls the projection light to display an interaction indicator in a preset command recognition area after the robot reaches the interaction point. This indicator instructs the user to initiate an interaction action in the corresponding area based on the interaction indicator. The robot then controls the visual sensor to collect data associated with the interaction action and identifies the collected data to obtain the human-machine interaction instruction. By identifying the information collected by the visual sensor to obtain the human-machine interaction instruction, the user does not need to manually operate the robot's touch screen and is not affected by ambient noise, thereby improving the efficiency and reliability of interaction instruction recognition.
[0099] Example 3
[0100] FIG3 is a robot interaction method provided in Example 3 of the present application. This embodiment can be applied to identify human-machine interaction instructions based on the collected data of the robot's visual sensor. The method can be applied to the robot's processor. As shown in FIG3, the method includes:
[0101] S310: When the robot moves toward the human-machine interaction point, abnormality detection is performed through the data collected by the visual sensor.
[0102] Optionally, the human-computer interaction point includes at least one of a delivery starting point and a delivery target point.
[0103] In this optional embodiment, in the scenario where the robot is a delivery robot, the human-machine interaction point can be at least one of a delivery starting point and a delivery target point.
[0104] S320: When the robot reaches the human-machine interaction point, suspend abnormality detection and open the storage compartment door.
[0105] In an embodiment of the present application, after the robot reaches the human-machine interaction point, it is necessary to place or remove the delivery item. At this time, anomaly detection is suspended and the storage compartment door is opened to facilitate the user's operation of the delivery item. Exemplarily, after the delivery robot responds to the user's call and arrives at the delivery starting point, anomaly detection is suspended, the storage compartment door is opened, and the user is waiting to place the delivery item. Another exemplary embodiment is that after the delivery robot reaches the delivery destination point, anomaly detection is suspended, the storage compartment door is opened, and the user is waiting to remove the delivery item.
[0106] S330: After detecting that the door is open and the object changes stably, identify the door closing instruction based on the data collected by the visual sensor.
[0107] In this embodiment of the present application, after the storage compartment door is opened, the changes in items within the storage compartment tray are detected in real time. If the items change (e.g., an increase or decrease in items) and remain stable within a set time period t after the change, i.e., no longer change, the door closing command is identified using data collected by the visual sensor. Specifically, the data collected by the visual sensor is received and converted from sensor data into point cloud data. The point cloud data is then converted into the robot coordinate system. The converted data features are then matched with pre-set door closing command features. If the two match, the door closing command is recognized.
[0108] In scenarios like food delivery, when using a robot, the user may perform multiple actions. For example, after placing dish A on the robot's first shelf, the user may need to place the dish on the second shelf before the robot can begin delivery. Therefore, determining the set duration t based on the robot's operating scenario can improve the robot's operational stability. Otherwise, if the robot recognizes a change in an object and begins to recognize a door-closing command, and the user steps on or is within the corresponding command recognition area while placing the dish, the robot may mistakenly recognize the action as a door-closing command and close the door, potentially causing accidental injury to the user or spilling food.
[0109] Exemplarily, after arriving at the delivery starting point, the robot opens the storage compartment door and waits for the user to place the item. If the robot detects that the user has placed the item in the storage compartment and that the item remains stable for 5 seconds, the robot begins recognizing user-initiated door-closing commands based on data collected by the visual sensor. In another exemplary embodiment, after arriving at the delivery destination, the robot opens the storage compartment door and waits for the user to remove the item. If the robot detects that the user has removed the item and that the item remains stable for 5 seconds, the robot begins recognizing user-initiated door-closing commands based on data collected by the visual sensor.
[0110] S340: Confirm the door closing command, close the storage compartment door, and control the robot to go to the next delivery target point or return to the origin.
[0111] In an embodiment of the present application, after the processor recognizes the door-closing instruction, it closes the storage compartment door and controls the robot to go to the next delivery target point or return to the origin. Exemplarily, after the robot reaches the delivery starting point and detects that the user places the items to be delivered and they remain unchanged, it recognizes the door-closing instruction. After confirming that the door-closing instruction is recognized, the storage compartment door is closed and the robot is controlled to go to the next delivery target point. Another exemplary embodiment, after the robot reaches the delivery target point and detects that the user takes out the items and the items in the storage compartment remain unchanged, it recognizes the door-closing instruction. After confirming that the door-closing instruction is recognized, it closes the storage compartment door and controls the robot to go to the next delivery target point or return to the origin.
[0112] The technical solution of the embodiment of this application opens the storage compartment door when the robot reaches the human-machine interaction point. After detecting that the compartment door is open and the contents are stable, the robot identifies the door-closing command based on the data collected by the visual sensor. Finally, the door-closing command is confirmed, closing the compartment door and controlling the robot to proceed to the next delivery destination or return to its starting point. By using the visual sensor to collect data and identify the door-closing command, the user no longer needs to manually touch the screen or initiate a voice command, improving the efficiency and stability of command recognition and, in turn, the robot's delivery efficiency.
[0113] Example 4
[0114] Figure 4 is a schematic diagram of the structure of a robot provided in Example 4 of the present application. This embodiment is suitable for identifying human-machine interaction instructions based on data collected by a visual sensor installed on the robot. The robot 400 includes a visual sensor 401 and a processor, and the processor and visual sensor 401 are in communication.
[0115] The visual sensor 401 is used to send collected data to the processor when the robot 400 moves toward the human-machine interaction point or reaches the human-machine interaction point.
[0116] In this embodiment of the present application, the visual sensor 401 is used to send collected data to the processor while the robot 400 is moving toward and upon reaching the human-machine interaction point. The data collected during movement toward the human-machine interaction point is used for anomaly detection, such as cliff and obstacle detection; the data collected after reaching the human-machine interaction point is used to identify human-machine interaction commands.
[0117] A processor is configured to execute the following robot interaction method:
[0118] During the movement of the robot 400 toward the human-machine interaction point, abnormality detection is performed through the data collected by the visual sensor 401;
[0119] When the robot 400 reaches the human-machine interaction point, abnormality detection is suspended;
[0120] The human-computer interaction instruction is identified based on the collected data of the visual sensor 401.
[0121] In this embodiment of the present application, a processor is installed within robot 400. After receiving data collected by visual sensor 401 while robot 400 is moving toward a human-machine interaction point, the processor processes the collected data to detect anomalies in the robot's motion path, such as obstacles or cliffs. After robot 400 reaches the human-machine interaction point, the anomaly detection process is suspended and the robot switches from anomaly detection mode to human-machine interaction command detection mode. In human-machine interaction command detection mode, the processor identifies human-machine interaction commands by analyzing the data collected by visual sensor 401.
[0122] Optionally, the robot 400 further includes a projection light 403 , which is communicatively connected to the processor;
[0123] The projection light 403 is used to display an interaction indication mark in a preset instruction recognition area, and to instruct the user to initiate an interaction action in the corresponding area based on the interaction indication mark.
[0124] Optionally, the visual sensor 401 is tilted downward, the installation position of the visual sensor 401 is 0.3-2.0 meters from the ground, the installation position of the projection lamp 403 is 0.1-1.0 meters from the ground, and the installation position of the visual sensor 401 is higher than the installation position of the projection lamp.
[0125] In this optional embodiment, the installation positions of the visual sensor 401 and the projector 403 are provided: the visual sensor 401 is installed in front of the robot 400 at a height of L1 from the ground. For example, L1 is 0.3-2.0 meters. For example, the robot 400 is 1.3 meters tall, and the visual sensor 401 is installed in front of the robot 400 at a height of 1-1.2 meters from the ground. In this solution, the visual sensor 401 is used to detect obstacles in the robot's motion path. To address the problem of users being unable to interact with the robot 400 via the touch screen due to their hands being occupied, this solution considers users initiating human-computer interaction commands via their feet. That is, the visual sensor 401 is also used to collect interaction commands initiated by the user via their feet. From the above analysis, it can be seen that the objects that the visual sensor 401 needs to detect are all close to the ground. Therefore, the visual sensor 401 is tilted downward. The tilt angle α is related to the installation height of the visual sensor 401. The purpose is to ensure that the lower edge of the visual sensor 401's field of view is connected to the front edge of the robot 400. For example, it is tilted downward and has an angle of 28°-33° with the direction of gravity.
[0126] Projector 403 is used to project an interactive indicator, which prompts the user to initiate a foot interaction command. Therefore, the interactive indicator is typically projected onto the ground in front of robot 400. Given the aforementioned projection requirements, projector 403 can be placed in front of robot 400 at a height L2 above the ground. For example, L2 can be 0.1-1.0 meters. To ensure that the interactive indicator projected by projector 403 is projected smoothly onto the ground without being obstructed by waiters taking food or opening and closing storage compartment doors, projector 403 can be placed at a lower height, for example, 0.2-0.3 meters above the ground. Projector 403 is mounted downwardly, with the tilt angle β being related to the installation height of projector 403 and the desired projection area. This can be determined based on the installation height of projector 403 and the desired projection location on the ground. For example, if projector 403 is mounted at a height L2 and the projection location is located directly in front of robot 400 at a distance S, the tilt angle is arctan(S / L2). The projection location on the ground must be within the field of view of visual sensor 401.
[0127] It is worth noting that the installation height of the stereo vision sensor 401 is greater than that of the projection lamp 403, which can expand the detection field of the stereo vision sensor 401. The projection lamp 403 is installed at a lower position to reduce the impact of robot operation and other obstacles on the projection.
[0128] For example, robot 400 is equipped with an interactive screen 402, and a stereo vision sensor 401 is positioned below interactive screen 402 to facilitate stable data collection in the area below and in front of the robot 400's movement direction. Robot 400 is also equipped with a laser radar 404, and a projector 403 is positioned above the laser radar 404 to avoid affecting the laser radar data collection.
[0129] The technical solution of the embodiment of the present application uses the data collected by the visual sensor to detect anomalies during the robot's movement toward the human-machine interaction point. When the robot reaches the human-machine interaction point, the anomaly detection is suspended and the human-machine interaction command is identified based on the data collected by the visual sensor. By processing the data collected by the visual sensor differently in different working states of the robot, human-machine interaction is achieved, avoiding the shortcomings of touch screen interaction, such as low interaction efficiency when both hands are occupied, and high error rate of voice interaction in noisy environments. This solution can improve the efficiency and reliability of human-machine interaction without increasing hardware costs.
[0130] Example 5
[0131] FIG5 is a schematic diagram of the structure of a robot interaction device provided in Example 5 of the present application. As shown in FIG5 , the device includes:
[0132] Anomaly detection module 510, used to detect anomalies using data collected by the visual sensor during the robot's movement toward the human-machine interaction point;
[0133] An abnormality detection suspension module 520 is used to suspend the abnormality detection when the robot reaches the human-machine interaction point;
[0134] The instruction recognition module 530 is used to recognize human-computer interaction instructions based on the data collected by the visual sensor.
[0135] The technical solution of the embodiment of the present application uses the data collected by the visual sensor to detect anomalies during the robot's movement toward the human-machine interaction point. When the robot reaches the human-machine interaction point, the anomaly detection is suspended and the human-machine interaction command is identified based on the data collected by the visual sensor. By processing the data collected by the visual sensor differently in different working states of the robot, human-machine interaction is achieved, avoiding the shortcomings of touch screen interaction, such as low interaction efficiency when both hands are occupied, and high error rate of voice interaction in noisy environments. This solution can improve the efficiency and reliability of human-machine interaction without increasing hardware costs.
[0136] Optionally, the instruction recognition module 530 includes:
[0137] A projection light control submodule, configured to control the projection light to display an interactive indicator in a preset instruction recognition area, and to instruct the user to initiate an interactive action in the corresponding area based on the interactive indicator;
[0138] A data acquisition submodule, configured to control the visual sensor to acquire data associated with the interactive action;
[0139] The interactive instruction recognition submodule is used to recognize the collected data and obtain human-computer interaction instructions.
[0140] Optionally, the visual sensor is tilted downward, the installation position of the visual sensor is 0.3-2.0 meters from the ground, the installation position of the projection lamp is 0.1-1.0 meters from the ground, and the installation position of the visual sensor is higher than the installation position of the projection lamp.
[0141] Optionally, the interactive instruction recognition submodule includes:
[0142] An initial point set acquisition unit, configured to convert the collected data into point cloud data, and convert the point cloud data into a robot coordinate system to obtain an initial point set;
[0143] an action point screening unit, configured to screen the initial point set to obtain interactive action points contained in the initial point set;
[0144] The interaction instruction determination unit is used to determine the human-computer interaction instruction by identifying the interaction action point.
[0145] Optional, action point screening unit, including:
[0146] a transition point set screening subunit, configured to screen out a transition point set located within the instruction recognition area from the initial point set;
[0147] The ground point filtering subunit is used to filter out ground points in the transition point set to obtain interactive action points.
[0148] Optionally, the interactive instruction determining unit includes:
[0149] an action point clustering subunit, configured to cluster the interactive action points according to the distances between the interactive action points to obtain at least one cluster;
[0150] a target cluster determination subunit, configured to determine, in the at least one cluster, a target cluster containing the most interactive action points, and determine a proportion of the target interactive action points in the target cluster among the interactive action points;
[0151] an interaction feature comparison subunit, configured to determine whether the target cluster conforms to a preset interaction feature based on the axial size, lateral size, and axial normal vector distribution of the target cluster when the number ratio is greater than a set ratio threshold;
[0152] The interaction instruction determination subunit is configured to determine a human-computer interaction instruction based on the target cluster when the target cluster matches the preset interaction feature.
[0153] Optionally, the ground point filtering subunit is specifically configured to perform at least one of the following:
[0154] According to a preset horizontal ground height range, ground points are filtered out at the transition points to obtain interactive action points;
[0155] The extracted ground parameters of the non-horizontal ground are converted to the robot coordinate system, and the ground points in the transition point concentration are filtered out according to the converted ground parameters to obtain the interactive action points.
[0156] Optionally, the interactive instruction determines a subunit, specifically for:
[0157] When the target cluster matches the preset interaction feature, obtaining the timing duration of the interaction detection timer; the interaction detection timer is used to time the duration of the identified interaction action;
[0158] When the timing duration reaches a set time threshold, a human-computer interaction instruction is determined according to the target cluster.
[0159] Optionally, the human-machine interaction point includes at least one of a delivery starting point and a delivery target point, the human-machine interaction instruction includes a door closing instruction, and the robot interaction device further includes:
[0160] A hatch opening module, configured to open the storage hatch when the robot reaches the human-machine interaction point;
[0161] The instruction recognition module 530 is further configured to:
[0162] After detecting that the door is open and the object changes stably, identifying the door closing instruction based on the collected data of the visual sensor;
[0163] If the door closing instruction is confirmed, the storage compartment door is closed, and the robot is controlled to go to the next delivery target point or return to the origin.
[0164] The robot interaction device provided in the embodiments of the present application can execute the robot interaction method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method.
[0165] Example 6
[0166] FIG6 shows a block diagram of an electronic device 10 that can be used to implement an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.
[0167] As shown in Figure 6, electronic device 10 includes at least one processor 11 and memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer programs stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of electronic device 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. An input / output (I / O) interface 15 is also connected to bus 14.
[0168] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.
[0169] Processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any other suitable processor, controller, microcontroller, etc. Processor 11 executes the various methods and processes described above, such as the robot interaction method.
[0170] In some embodiments, the robot interaction method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the robot interaction method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the robot interaction method in any other suitable manner (e.g., via firmware).
[0171] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0172] Computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0173] In the context of the present application, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0174] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0175] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0176] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.
[0177] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of this application can be achieved. This is not limited herein.
[0178] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.
Claims
1. A robot interaction method, It is characterized in that A processor applied to a robot, wherein the robot is provided with a visual sensor, and the method comprises: When the robot moves toward the human-machine interaction point, abnormality detection is performed through the collected data of the visual sensor; When the robot reaches the human-machine interaction point, suspending the abnormality detection; The human-computer interaction instruction is identified according to the collected data of the visual sensor.
2. The method according to claim 1, It is characterized in that The robot is also provided with a projection lamp, and the identifying of the human-machine interaction instruction according to the collected data of the visual sensor comprises: Controlling the projection light to display an interactive indication mark in a preset command recognition area, so as to instruct the user to initiate an interactive action in a corresponding area based on the interactive indication mark; Controlling the visual sensor to collect data associated with the interactive action; The collected data is identified to obtain a human-computer interaction instruction.
3. The method according to claim 2, It is characterized in that The visual sensor is tilted downward, the installation position of the visual sensor is 0.3-2.0 meters from the ground, the installation position of the projection lamp is 0.1-1.0 meters from the ground, and the installation position of the visual sensor is higher than the installation position of the projection lamp.
4. The method according to claim 2, It is characterized in that The step of identifying the collected data to obtain a human-computer interaction instruction includes: Converting the collected data into point cloud data, and converting the point cloud data into a robot coordinate system to obtain an initial point set; Screening the initial point set to obtain interactive action points included in the initial point set; The human-computer interaction instruction is determined by identifying the interaction action point.
5. The method according to claim 4, It is characterized in that The screening of the initial point set to obtain the interactive action points contained in the initial point set includes: In the initial point set, a transition point set located in the instruction recognition area is selected; The ground points are filtered out in the transition point set to obtain the interactive action points.
6. The method according to claim 4, It is characterized in that The step of identifying the interaction action point to determine the human-computer interaction instruction includes: Clustering the interaction action points according to the distances between the interaction action points to obtain at least one cluster; In the at least one cluster, determine a target cluster containing the most interactive action points, and determine a ratio of the number of target interactive action points in the target cluster to the interactive action points; When the number ratio is greater than a set ratio threshold, determining whether the target cluster is consistent with the preset interaction feature according to the axial size, lateral size and axial normal vector distribution state of the target cluster; In the case that the target cluster matches the preset interaction feature, a human-computer interaction instruction is determined according to the target cluster.
7. The method according to claim 5, It is characterized in that The step of filtering out ground points from the transition point set to obtain interactive action points includes at least one of the following: According to a preset horizontal ground height range, ground points are filtered out at the transition points to obtain interactive action points; The extracted ground parameters of the non-horizontal ground are converted to the robot coordinate system, and the ground points in the transition point set are filtered out according to the converted ground parameters to obtain the interactive action points.
8. The method according to claim 6, It is characterized in that The step of determining the human-computer interaction instruction according to the target cluster includes: When the target cluster matches the preset interaction feature, obtaining the timing duration of the interaction detection timer; the interaction detection timer is used to time the duration of the identified interaction action; When the timing duration reaches a set time threshold, a human-computer interaction instruction is determined according to the target cluster.
9. The method according to claim 1, It is characterized in that The human-computer interaction point includes at least one of a delivery starting point and a delivery target point, the human-computer interaction instruction includes a door closing instruction, and the method further includes: When the robot reaches the human-machine interaction point, opening the storage compartment door; The identifying of the human-computer interaction instruction according to the collected data of the visual sensor includes: After detecting that the door is opened and the object changes stably, identifying the door closing instruction according to the collected data of the visual sensor; If the door closing instruction is confirmed, the storage compartment door is closed, and the robot is controlled to go to the next delivery target point or return to the origin.
10. A robot, It is characterized in that include: Vision sensors and processors; The processor is communicatively connected to the visual sensor; The visual sensor is used to send the collected data to the processor when the robot moves toward the human-machine interaction point and when it reaches the human-machine interaction point; The processor is used to execute the robot interaction method according to any one of claims 1 to 9.
11. A robot interaction device, It is characterized in that include: The anomaly detection module is used to detect anomalies through the data collected by the visual sensor when the robot moves to the human-machine interaction point; An abnormality detection suspension module is used to suspend the abnormality detection when the robot reaches the human-machine interaction point; The instruction recognition module is used to recognize human-computer interaction instructions based on the collected data of the visual sensor.
12. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the robot interaction method according to any one of claims 1 to 9 when executed.
Citation Information
Patent Citations
Method and machinery equipment for mobile goods selling
CN109767556A
Autonomous mobile device, door opening and closing control method and device and storage medium
CN114516046A
System and method for man-machine cooperation and man-machine collision prevention based on infrared camera and positioning recognition pad
CN114706403A
Wall surface quality detection robot and method, electronic equipment and storage medium
CN114995414A
Obstacle recognition method and device and electronic equipment
CN115273018A
Cited By
Industrial robot anomaly detection method based on recursive feature elimination and auto-encoder
CN119610214A
Engineering robot fault prediction method and system
CN121650024A
