Human-computer interaction method, device, storage medium, program product and robot

Abstract

The present application relates to a human-computer interaction method, device, robot, storage medium and computer program product, wherein the method includes: locating a desktop area; projecting a preset item selection interface onto the desktop area; identifying the click position of the user on the desktop area based on plane fitting technology; performing position conversion on the click position to determine the selected item. During the whole process, the preset item selection interface is displayed to the user in a projection manner for the user to select, and the selected item is determined based on the plane fitting technology, without relying on fixed human-computer interaction equipment, and can be flexibly applied to different application scenarios, bringing convenience to users.

Description

Technical Field

[0001] The present application relates to the field of intelligent interaction technology, and in particular to a human-computer interaction method, device, storage medium, computer program product, and robot. Background Art

[0002] With the development of science and technology, intelligent interaction technology has emerged. In the application scenarios of intelligent interaction technology, users (people) can interact with machines. Through these interactions, users can control machines to perform required operations.

[0003] In traditional technologies, this type of interaction is mostly based on display screen or voice control interactions, such as displaying a user-selectable menu on the display screen, and the user clicking on the display screen to order food, or the user using voice control to control the device to increase or decrease the volume or change songs.

[0004] Although the above-mentioned interaction methods can realize human-computer interaction and bring a certain degree of convenience to users, in actual applications, users' needs are diverse. The above-mentioned human-computer interaction methods that rely on fixed equipment cannot be applied to different application scenarios. Summary of the invention

[0005] Based on this, it is necessary to provide a human-computer interaction method, device, storage medium, computer program product and robot that can be flexibly applied to different application scenarios in response to the above technical problems.

[0006] In a first aspect, the present application provides a human-computer interaction method. The method comprises:

[0007] Locate the desktop area;

[0008] Projecting a preset item selection interface onto the desktop area;

[0009] Identifying a click position in the desktop area based on a plane fitting technique;

[0010] The click position is converted to determine the selected item.

[0011] In one embodiment, the locating the desktop area includes:

[0012] Collect environmental image data;

[0013] According to the environmental image data, identifying a preset logo set on the desktop;

[0014] The desktop area is located according to the preset identifier.

[0015] In one embodiment, the identifying the click position in the desktop area based on the plane fitting technology includes:

[0016] Performing a three-dimensional scan on the desktop area to establish a three-dimensional point cloud of the desktop;

[0017] Performing plane fitting on the desktop three-dimensional point cloud to extract a difference three-dimensional point cloud;

[0018] Identify point cloud data matching the preset click point cloud features in the difference three-dimensional point cloud to obtain target point cloud data;

[0019] A click position in the desktop area is identified according to the target point cloud data.

[0020] In one embodiment, identifying point cloud data in the difference three-dimensional point cloud that matches a preset click point cloud feature to obtain target point cloud data includes:

[0021] Get the preset click point cloud corresponding to the click posture;

[0022] The difference three-dimensional point cloud is feature matched with a preset click point cloud corresponding to the click posture to obtain target point cloud data.

[0023] In one embodiment, the item is a dish, and the table area is provided with a dining area;

[0024] After converting the click position to determine that the item has been selected, the method further includes:

[0025] Obtain the size of the dining area and locate the relative position relationship between the projection light source and the dining area;

[0026] Determining the required projection screen size according to the dining area size and the preset projection screen size conversion ratio;

[0027] According to the required projection screen size and the relative position relationship, the screen of the selected dish is projected into the dining area.

[0028] In one embodiment, after converting the click position to determine that the item has been selected, the method further includes:

[0029] Get the delivery path and path width of the selected item;

[0030] Projecting the delivery path and path width onto the ground;

[0031] When an obstacle is detected during driving, an image classification algorithm is used to identify the type of obstacle;

[0032] According to the obstacle type, an obstacle avoidance reminder message is pushed.

[0033] In a second aspect, the present application also provides a human-computer interaction device. The device comprises:

[0034] A positioning module, used for positioning the desktop area;

[0035] A first projection module, used for projecting a preset item selection interface onto the desktop area;

[0036] A fitting module, used for identifying a click position in the desktop area based on a plane fitting technology;

[0037] The position conversion module is used to convert the click position to determine the selected item.

[0038] In a third aspect, the present application further provides a robot, comprising a robot body and a processor, wherein the processor is disposed in the robot body and loaded with a computer program, and when the processor executes the computer program, the following steps are implemented:

[0039] Locate the desktop area;

[0040] Projecting a preset item selection interface onto the desktop area;

[0041] Identifying a click position in the desktop area based on a plane fitting technique;

[0042] The click position is converted to determine the selected item.

[0043] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0044] Locate the desktop area;

[0045] Projecting a preset item selection interface onto the desktop area;

[0046] Identifying a click position in the desktop area based on a plane fitting technique;

[0047] The click position is converted to determine the selected item.

[0048] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0049] Locate the desktop area;

[0050] Projecting a preset item selection interface onto the desktop area;

[0051] Identifying a click position in the desktop area based on a plane fitting technique;

[0052] The click position is converted to determine the selected item.

[0053] In actual application, the above-mentioned human-computer interaction method, device, robot, storage medium and computer program product first locate the desktop area; project the preset item selection interface to the desktop area; identify the click position in the desktop area based on the plane fitting technology; then convert the position of the click position to finally determine the selected item. In the whole process, the preset item selection interface is displayed to the user in a projection manner for the user to select, and the selected item is determined based on the plane fitting technology. It does not need to rely on fixed human-computer interaction equipment, can be flexibly applied to different application scenarios, and brings convenience to users. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 An application environment diagram of a human-computer interaction method in an embodiment;

[0055] Figure 2 A schematic diagram of a flow chart of a human-computer interaction method in an embodiment;

[0056] Figure 3 is a flowchart of a human-computer interaction method in another embodiment;

[0057] Figure 4 is a schematic diagram of a sub-process of S300 in an embodiment;

[0058] Figure 5 is a structural block diagram of a human-computer interaction device in one embodiment;

[0059] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] The human-computer interaction method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Specifically applied to the intelligent ordering scene, the intelligent ordering interactive system executes the human-computer interaction method of the application. The entire intelligent ordering interactive system has the functions of photography and projection, which can be specifically composed of a camera, a projection component and a control software. The camera can select an RGBD camera. The intelligent ordering interactive system can be set at the top in front of the robot, projecting 45 degrees downward, and the field of view of the camera component and the projection component are close. In actual application, the intelligent ordering interactive system collects environmental images, locates the desktop (dining table) area based on the collected environmental images, and projects the preset dish selection interface to the desktop area. The user orders dishes based on his own needs. The intelligent ordering interactive system identifies the user's click position in the desktop area based on the plane fitting technology, converts the click position, and determines the selected dish. For example, the user orders scrambled eggs with tomatoes. At this time, the intelligent ordering interactive system projects the picture of the user's selection of dishes to the desktop area, that is, the dish of scrambled eggs with tomatoes is projected to the desktop area, and can be further specifically projected into the plate on the desktop to display the final dish form in an image, giving the user a good experience. It should be noted that the human-computer interaction method can also be applied to user product selection, such as selecting toys, models, clothes, shoes and socks, and snacks. It is understandable that in another specific application example, the above-mentioned intelligent ordering interactive system can also be composed of a camera, a projection component, and a processor loaded with control software, and the realization of the entire intelligent ordering function is similar to the above content, which will not be repeated here.

[0062] In one embodiment, Figure 2 As shown, a human-computer interaction method is provided, comprising the following steps:

[0063] S100: Locate the desktop area.

[0064] The desktop area is an area used for projecting display images and objects. Taking the intelligent ordering interaction as an example, the desktop area is the dining table area. The specific intelligent human-computer interaction system can collect environmental images, analyze the environmental images, and identify and locate the desktop area. Furthermore, it is also possible to pre-set identifiers such as QR codes, bar codes, etc. on the table. After the intelligent human-computer interaction system collects the environmental images, it uses these identifiers based on the collected images to quickly locate the desktop area.

[0065] S200: Projecting a preset item selection interface onto a desktop area.

[0066] The preset item selection interface is a pre-configured interface. The interface configuration data can be pre-loaded into the intelligent human-computer interaction system and supports online updates. The preset item selection interface will display items that users can choose from. Taking the intelligent ordering scenario as an example, the preset object selection interface will display different dishes for users to choose from. The intelligent human-computer interaction system projects the preset item selection interface to the desktop area. When users browse to the interface, they can know the objects they can choose from.

[0067] S300: Identify a click position in the desktop area based on a plane fitting technology.

[0068] Plane fitting technology refers to identifying areas with large differences through plane fitting, and then identifying the user's click position in the desktop area. Specifically, a three-dimensional point cloud of the desktop can be established. At this time, there may be other objects on the desktop. The three-dimensional point cloud of the desktop is plane-fitted to extract point clouds with large differences. This part of the difference point cloud may be the point cloud corresponding to the object on the desktop or the point cloud corresponding to the user clicking a specific position on the desktop area (for example, the user's hand clicking a specific position). Then, the target point cloud data corresponding to the user clicking the desktop area is identified through feature matching, and the click position in the desktop area is identified based on this part of the target point cloud data.

[0069] S400: Convert the clicked position to determine the selected item.

[0070] After determining the single click position of the user in S300, the position of the click position is converted in the preset item selection interface to determine the selected item. For example, S300 determines that the coordinates of the click position of the user are (a, b), and the position of the click position is converted to determine that the click position is the position corresponding to the second left item in the second row in the preset item selection interface, and determines that the selected item is the second left item in the second row in the preset item selection interface.

[0071] The above human-computer interaction method first locates the desktop area; projects the preset item selection interface onto the desktop area; identifies the click position on the desktop area based on the plane fitting technology; and then converts the click position to finally determine the selected item. In the whole process, the preset item selection interface is displayed to the user by projection for selection, and the selected item is determined based on the plane fitting technology. It does not need to rely on fixed human-computer interaction equipment, and can be flexibly applied to different application scenarios, bringing convenience to users.

[0072] Furthermore, after determining that the object has been selected, the image of the selected object may be projected onto the desktop area.

[0073] Specifically, after determining that the item has been selected, the item selected by the user is displayed to the user who selected the item, giving the user a good human-computer interaction experience. Here, the screen of the user selecting the item is projected onto the desktop area. When projecting, you can choose to project to a specific location in the desktop area, such as an item carrier placed in the desktop area. Taking the smart ordering scenario as an example, after S400 determines that the user has selected a dish, the screen of the selected item and dish can be projected onto the plate on the dining table, so as to show the user the appearance of the ordered plate in an image, giving the user a good human-computer interaction experience. Figure 3 As shown, in one embodiment, S100 includes:

[0074] S120: Collecting environmental image data;

[0075] S140: Identify a preset logo set on the desktop according to the environmental image data;

[0076] S160: Locate the desktop area according to the preset identifier.

[0077] Environmental image data can be collected by an image acquisition device, for example, by a camera. As one of the embodiments, an RGBD camera can be used. A preset identifier refers to a preset identifier, which can specifically be a QR code, a barcode, and a specific graphic that serves as an identifier. The preset identifier is set on the desktop, so when the preset identifier is identified, the desktop area can be located. After the environmental image data is collected, the environmental image data is image processed to identify the preset identifier, and then the desktop area is located. Taking the smart ordering scene as an example, the smart ordering interactive system collects environmental images, searches for a QR code pre-attached to the desktop in the environmental image, and locates the tabletop area based on the position of the QR code. Furthermore, when multiple preset identifiers are searched and identified, the preset identifier with the closest distance is used as the valid preset identifier to locate the desktop area.

[0078] like Figure 4 As shown, in one embodiment, S300 includes:

[0079] S320: Perform three-dimensional scanning on the desktop area to create a three-dimensional point cloud of the desktop;

[0080] S340: performing plane fitting on the desktop 3D point cloud to extract the difference 3D point cloud;

[0081] S360: Identify point cloud data matching the preset click point cloud features in the difference three-dimensional point cloud to obtain target point cloud data;

[0082] S380: Identify a click position in the desktop area according to the target point cloud data.

[0083] The intelligent human-computer interaction system can perform a three-dimensional scan of the desktop area by using an RGBD camera to perform a three-dimensional scan and establish a desktop three-dimensional point cloud. At this time, there may be other objects on the desktop. The existence of these objects will be characterized as the difference of the three-dimensional point cloud. Therefore, the desktop three-dimensional point cloud can be plane-fitted to extract the difference three-dimensional point cloud. The specific fitting process can be that when the point cloud distance from the fitting plane exceeds a preset distance threshold (for example, more than 1 cm), it is considered that there is an object on the desktop. By extracting the area with large differences, a difference three-dimensional point cloud can be obtained. The difference three-dimensional point cloud represents the height and shape of the object on the desktop. The object on the desktop may include a click part (for example, a user's hand) where the user clicks on the desktop area. In order to identify the position where the user's hand clicks, it is necessary to first identify the point cloud data of the user's hand in the difference three-dimensional point cloud by feature matching, and identify the user's click position in the desktop area by analyzing this part of the target point cloud data. Specifically, the pre-collected preset click point cloud and the difference three-dimensional point cloud can be feature-matched to obtain the target point cloud data. The pre-collected preset click point cloud is the point cloud data corresponding to the preset gesture action. The preset gesture is specifically a pre-set gesture action. The gesture action can be pre-set according to the user's habits or application scenarios, and prompt the user. For example, the gesture action can be the action of extending the index finger. It can be understood that the gesture action can also be the action of extending the thumb, or it can be directly the action of extending the finger.

[0084] In one embodiment, identifying point cloud data matching a preset click point cloud feature in a difference three-dimensional point cloud to obtain target point cloud data includes:

[0085] The preset click point cloud corresponding to the click posture is obtained; the difference three-dimensional point cloud is feature matched with the preset click point cloud corresponding to the click posture to obtain target point cloud data. The preset click point cloud corresponding to the click posture is pre-collected and stored, and specifically, the preset click point cloud corresponding to the index finger gesture can be extended.

[0086] In one embodiment, the item is a dish, and the table area is provided with a dining area;

[0087] After converting the click position and confirming that the item has been selected, it also includes:

[0088] Obtain the size of the dining area, and locate the relative position relationship between the projection light source and the dining area; determine the required projection screen size based on the size of the dining area and the preset projection screen size conversion ratio; project the screen of the selected dish to the dining area based on the required projection screen size and the relative position relationship. The dining area refers to the area on the desktop where users place dishes / foods, which can specifically be the area on the desktop where dinner plates are placed. The projection light source refers to the light source that emits the projection screen, and the relative position between the projection light source and the dining area is located so that the screen of the selected dish can be accurately projected to the dining area later. Specifically, this projection, on the one hand, determines the projection angle based on the relative position relationship between the projection light source and the dining area, and on the other hand, adjusts the size of the screen projected to the dining area based on the required projection screen size.

[0089] In this embodiment, the human-computer interaction method of the application is applied to the intelligent ordering scenario. The user needs to select the required dishes from the projected menu screen. After the user completes the dish selection (orders the dishes), the intelligent ordering interaction system scans and obtains the size of the plate on the table, and locates the relative posture relationship between itself and the plate, and calculates the required projection screen size according to the size of the plate. This calculation can be specifically based on the plate size. The projection screen size is converted according to a preset ratio. For example, the projection screen size can be 85% of the plate size. Based on the relative posture relationship between the projection light source itself and the plate, the dish image after size transformation is projected into the plate to vividly display the dish, giving the user a good human-computer interaction experience.

[0090] In one embodiment, after converting the click position to determine that the item has been selected, the method further includes:

[0091] Obtain the delivery path and path width of the selected items; project the delivery path and path width onto the ground; when an obstacle is detected during travel, use image classification algorithm to identify the type of obstacle; push obstacle avoidance reminder messages based on the type of obstacle.

[0092] After the intelligent human-computer interaction system projects the item screen of the selected item onto the desktop area, it begins to enter the item delivery stage, first obtaining the item delivery route data, wherein the item delivery route data specifically includes the delivery path and the path width, which can be specifically generated by a pre-loaded path planning program. After obtaining the delivery path and the path width, in order to prompt pedestrians on the delivery path to avoid collisions, the delivery path and the path width are projected onto the ground, and when the intelligent human-computer interaction system detects an obstacle during its movement, it uses an image classification algorithm to identify the type of obstacle, and pushes a targeted obstacle reminder message according to the type of obstacle. Specifically, if the obstacle is a person, a reminder message can be pushed to prompt the person to avoid it; if the obstacle is a fixed, immovable object, such as a table, chair, or bench, a reminder message to modify the position of the table, chair, or bench can be pushed to the staff, specifically to the staff's wristband; if it is other obstacles, a preset conspicuous image can be projected on the obstacle, and a clear reminder message can be pushed to the staff.

[0093] In order to explain the above-mentioned technical solution of obstacle avoidance reminder in detail, the following will provide a detailed and complete description of the smart ordering scenario.

[0094] The robot is loaded with an intelligent ordering interactive system. After the user completes the order, the robot enters the dish delivery (serving) stage. During the robot's operation, in order to prompt nearby pedestrians to avoid collision with the robot, the path and path width that the robot will take can be projected onto the ground through a projector. If an obstacle is found on the planned path during the movement, it means that this obstacle appeared after the map was established and is a new obstacle. First, the color camera combined with the image classification algorithm is used to identify the type of obstacle. If it is a pedestrian, the pedestrian is prompted to avoid it through voice. If the obstacle is identified as a table, chair or bench, it is determined whether there is a new passable path. If there is a passable path, the path planning is updated with the passable path. If there is no passable path, the waiter is notified to modify the position of the table and chair (the waiter wears a bracelet connected to the machine, and the machine automatically sends a message to the bracelet) to ensure that the robot can operate normally. If the obstacle is other obstacles, such as garbage or other unknown objects, a more conspicuous pattern will be projected on the obstacle, and the waiter will be notified to clear the obstacle.

[0095] To explain the technical solution of the human-computer interaction method in detail, the following will take the human-computer interaction method in the restaurant ordering scenario as an example to explain the whole process in detail. In the restaurant ordering scenario, the human-computer interaction method in the restaurant ordering scenario includes the following steps:

[0096] 1. Post the QR code on the dining table.

[0097] 2. When the color camera recognizes the QR code, it recognizes the table, and the projector projects the menu screen.

[0098] 3. The RGB camera performs a 3D scan on the dining table area identified by the color camera and creates a 3D point cloud of the desktop. There may be other objects on the desktop at this time. The 3D point cloud on the desktop is plane fitted. When the point cloud is more than 1 cm away from the fitting plane, it is considered that there is an object on the desktop. By extracting the area with large differences (the above 1 cm standard), the height and shape of the object on the desktop can be obtained.

[0099] 4. By matching the object point cloud with the pre-collected hand point cloud (the preset finger action is to only extend the index finger), it is possible to determine whether the object is a hand. Then, by calculating the height of the finger from the desktop, it is determined whether the finger touches the desktop, and the screen coordinate system coordinates corresponding to the touch point are determined by position conversion.

[0100] 5. Send click commands to realize the function of ordering food by clicking on the desktop.

[0101] 6. Match the object on the table with the pre-recorded plate point cloud (multiple sets are supported). If the match is successful, it means that the object is a plate. When the order is completed, the dish pattern is projected onto the plate. The specific steps are: according to the identified plate type, calculate the size of the screen to be projected, the screen size is 85% of the plate size, and according to the relative position relationship between the projector and the plate, calculate the transformed image and project it onto the plate.

[0102] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0103] Based on the same inventive concept, the embodiment of the present application also provides a human-computer interaction device for implementing the human-computer interaction method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more human-computer interaction device embodiments provided below can refer to the limitations on the human-computer interaction method above, and will not be repeated here.

[0104] In one embodiment, Figure 5 As shown, a human-computer interaction device is provided, comprising:

[0105] A positioning module 100, used for positioning a desktop area;

[0106] The projection module 200 is used to project the preset item selection interface onto the desktop area;

[0107] A fitting module 300, for identifying a click position in a desktop area based on a plane fitting technique;

[0108] The position conversion module 400 is used to convert the click position to determine the selected item.

[0109] The above-mentioned human-computer interaction device first locates the desktop area; projects the preset item selection interface onto the desktop area; identifies the click position on the desktop area based on the plane fitting technology; and then converts the click position to finally determine the selected item. During the whole process, the preset item selection interface is displayed to the user in a projection manner for the user to select, and the selected item is determined based on the plane fitting technology. It does not need to rely on fixed human-computer interaction equipment, and can be flexibly applied to different application scenarios, bringing convenience to users.

[0110] In one of the embodiments, the positioning module is further used to collect environmental image data; identify a preset mark set on the desktop based on the environmental image data; and locate the desktop area based on the preset mark.

[0111] In one of the embodiments, the fitting module is also used to perform a three-dimensional scan of the desktop area to establish a three-dimensional point cloud of the desktop; perform plane fitting on the three-dimensional point cloud of the desktop to extract a difference three-dimensional point cloud; identify point cloud data in the difference three-dimensional point cloud that matches a preset click point cloud feature to obtain target point cloud data; and identify the click position in the desktop area based on the target point cloud data.

[0112] In one of the embodiments, the fitting module is further used to obtain a preset click point cloud corresponding to the click posture; perform feature matching on the difference three-dimensional point cloud and the preset click point cloud corresponding to the click posture to obtain target point cloud data.

[0113] In one of the embodiments, the item is a dish, and a dining area is provided in the desktop area; the human-computer interaction device further comprises a second projection module, and the second projection module is used to project the object image of the selected item onto the desktop area, including: obtaining the size of the dining area, and locating the relative posture relationship between the projection light source and the dining area; determining the required projection image size according to the size of the dining area and a preset projection image size conversion ratio; and projecting the image of the selected dish onto the dining area according to the required projection image size and the relative posture relationship.

[0114] In one of the embodiments, the human-computer interaction device further includes an obstacle avoidance module for obtaining the delivery path and path width of the selected item; projecting the delivery path and path width onto the ground; using an image classification algorithm to identify the type of obstacle when an obstacle is detected during travel; and pushing an obstacle avoidance reminder message based on the type of obstacle.

[0115] Each module in the above-mentioned human-computer interaction device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0116] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a human-computer interaction method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0117] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] In one embodiment, a robot is provided, including a robot body and a processor, wherein the processor is disposed in the robot body and loaded with a computer program, and when the processor executes the computer program, the following steps are implemented:

[0119] Locate the desktop area;

[0120] Project the preset item selection interface to the desktop area;

[0121] Identify the click position in the desktop area based on plane fitting technology;

[0122] Convert the clicked position to determine the selected item.

[0123] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0124] Collect environmental image data; identify a preset mark set on the desktop based on the environmental image data; and locate the desktop area based on the preset mark.

[0125] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0126] Perform a three-dimensional scan on the desktop area to establish a three-dimensional point cloud of the desktop; perform plane fitting on the three-dimensional point cloud of the desktop to extract a difference three-dimensional point cloud; identify point cloud data in the difference three-dimensional point cloud that matches the preset click point cloud features to obtain target point cloud data; and identify the click position in the desktop area based on the target point cloud data.

[0127] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0128] Obtain a preset click point cloud corresponding to the click posture; perform feature matching on the difference three-dimensional point cloud and the preset click point cloud corresponding to the click posture to obtain target point cloud data.

[0129] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0130] Obtain the size of the dining area and locate the relative position relationship between the projection light source and the dining area; determine the required projection screen size based on the size of the dining area and the preset projection screen size conversion ratio; project the screen of the selected dish onto the dining area based on the required projection screen size and the relative position relationship.

[0131] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0132] Obtain the delivery path and path width of the selected items; project the delivery path and path width onto the ground; when an obstacle is detected during travel, use image classification algorithm to identify the type of obstacle; push obstacle avoidance reminder messages based on the type of obstacle.

[0133] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0134] Locate the desktop area;

[0135] Project the preset item selection interface to the desktop area;

[0136] Identify the click position in the desktop area based on plane fitting technology;

[0137] Convert the clicked position to determine the selected item.

[0138] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0139] Collect environmental image data; identify a preset mark set on the desktop based on the environmental image data; and locate the desktop area based on the preset mark.

[0140] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0141] Perform a three-dimensional scan on the desktop area to establish a three-dimensional point cloud of the desktop; perform plane fitting on the three-dimensional point cloud of the desktop to extract a difference three-dimensional point cloud; identify point cloud data in the difference three-dimensional point cloud that matches the preset click point cloud features to obtain target point cloud data; and identify the click position in the desktop area based on the target point cloud data.

[0142] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0143] Obtain a preset click point cloud corresponding to the click posture; perform feature matching on the difference three-dimensional point cloud and the preset click point cloud corresponding to the click posture to obtain target point cloud data.

[0144] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0145] Obtain the size of the dining area and locate the relative position relationship between the projection light source and the dining area; determine the required projection screen size based on the size of the dining area and the preset projection screen size conversion ratio; project the screen of the selected dish onto the dining area based on the required projection screen size and the relative position relationship.

[0146] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0147] Obtain the delivery path and path width of the selected items; project the delivery path and path width onto the ground; when an obstacle is detected during travel, use image classification algorithm to identify the type of obstacle; push obstacle avoidance reminder messages based on the type of obstacle.

[0148] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0149] Locate the desktop area;

[0150] Project the preset item selection interface to the desktop area;

[0151] Identify the click position in the desktop area based on plane fitting technology;

[0152] Convert the clicked position to determine the selected item.

[0153] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0154] Collect environmental image data; identify a preset mark set on the desktop based on the environmental image data; and locate the desktop area based on the preset mark.

[0155] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0156] Perform a three-dimensional scan on the desktop area to establish a three-dimensional point cloud of the desktop; perform plane fitting on the three-dimensional point cloud of the desktop to extract a difference three-dimensional point cloud; identify point cloud data in the difference three-dimensional point cloud that matches the preset click point cloud features to obtain target point cloud data; and identify the click position in the desktop area based on the target point cloud data.

[0157] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0158] Obtain a preset click point cloud corresponding to the click posture; perform feature matching on the difference three-dimensional point cloud and the preset click point cloud corresponding to the click posture to obtain target point cloud data.

[0159] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0160] Obtain the size of the dining area and locate the relative position relationship between the projection light source and the dining area; determine the required projection screen size based on the size of the dining area and the preset projection screen size conversion ratio; project the screen of the selected dish onto the dining area based on the required projection screen size and the relative position relationship.

[0161] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0162] Obtain the delivery path and path width of the selected items; project the delivery path and path width onto the ground; when an obstacle is detected during travel, use image classification algorithm to identify the type of obstacle; push obstacle avoidance reminder messages based on the type of obstacle.

[0163] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0164] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0165] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A human-computer interaction method, characterized in that: The method comprises: Locate the desktop area; Projecting a preset item selection interface onto the desktop area; Identifying a click position in the desktop area based on a plane fitting technique; Convert the clicked position to determine the selected item; The method of identifying the click position in the desktop area based on the plane fitting technology includes: performing a three-dimensional scan on the desktop area to establish a three-dimensional point cloud of the desktop; performing plane fitting on the three-dimensional point cloud of the desktop to extract a difference three-dimensional point cloud; identifying point cloud data in the difference three-dimensional point cloud that matches a preset click point cloud feature to obtain target point cloud data; and identifying the click position in the desktop area based on the target point cloud data.

2. The method according to claim 1, characterized in that The positioning desktop area includes: Collect environmental image data; According to the environmental image data, identifying a preset logo set on the desktop; The desktop area is located according to the preset identifier.

3. The method according to claim 1, characterized in that The step of identifying point cloud data in the difference three-dimensional point cloud that matches the preset click point cloud feature to obtain target point cloud data comprises: Get the preset click point cloud corresponding to the click posture; The difference three-dimensional point cloud is feature matched with a preset click point cloud corresponding to the click posture to obtain target point cloud data.

4. The method according to claim 1, characterized in that: The item is a dish, and the table area is provided with a dining area; after converting the click position and determining that the item has been selected, the method further includes: Obtain the size of the dining area and locate the relative position relationship between the projection light source and the dining area; Determining the required projection screen size according to the dining area size and the preset projection screen size conversion ratio; According to the required projection screen size and the relative position relationship, the screen of the selected dish is projected to the dining area.

5. The method according to claim 1, characterized in that After converting the click position to determine that the item has been selected, the method further includes: Get the delivery path and path width of the selected item; Projecting the delivery path and path width onto the ground; When an obstacle is detected during driving, an image classification algorithm is used to identify the type of obstacle; According to the obstacle type, an obstacle avoidance reminder message is pushed.

6. A human-computer interaction device, characterized in that: The device comprises: A positioning module, used for positioning the desktop area; A projection module, used for projecting a preset item selection interface onto the desktop area; A fitting module, used for identifying a click position in the desktop area based on a plane fitting technology; A position conversion module, used to convert the click position to determine the selected item; The fitting module is also used to perform a three-dimensional scan of the desktop area to establish a three-dimensional point cloud of the desktop; perform plane fitting on the desktop three-dimensional point cloud to extract a difference three-dimensional point cloud; identify point cloud data in the difference three-dimensional point cloud that matches a preset click point cloud feature to obtain target point cloud data; and identify a click position in the desktop area based on the target point cloud data.

7. The device according to claim 6, characterized in that The positioning module is also used to collect environmental image data; identify a preset mark set on the desktop according to the environmental image data; and locate the desktop area according to the preset mark.

8. A robot, characterized in that: The method comprises a robot body and a processor, wherein the processor is arranged on the robot body and loaded with a computer program, and the processor implements the steps of the method according to any one of claims 1 to 5 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images