Robot-based beverage ordering method, device and electronic equipment

Abstract

The application relates to a robot-based beverage ordering method and device and electronic equipment, comprising: acquiring multi-modal interaction information and user information of a user; the multi-modal interaction information respectively has a corresponding weight coefficient; acquiring ordering environment information of the user; adjusting the weight coefficient of the multi-modal interaction information according to the ordering environment information; determining recommended beverage information according to the multi-modal interaction information with the adjusted weight coefficient and the user information of the user; and recommending the recommended beverage information to the user for ordering according to the recommended beverage information. The embodiment of the application can adaptively adjust the weight coefficient of the multi-modal interaction information according to the ordering environment information, avoid the influence of the ordering environment information on the accuracy of understanding the user's intention, accurately understand the user's intention by combining the multi-modal interaction information with the adjusted weight coefficient and the user information, determine recommended beverage information meeting the user's demand, and improve the user's ordering efficiency and the user's robot-based ordering experience.

Description

Technical Field

[0001] This application belongs to the field of computer science, and specifically relates to a robot-based beverage ordering method, apparatus, and electronic device. Background Technology

[0002] Currently, the ordering method of robots mainly relies on touch screens, allowing users to place their orders independently by viewing the displayed beverage menu.

[0003] However, due to the limited interaction methods between users and robots, the robots struggle to accurately understand user intentions when placing orders. This makes them unfriendly to special user groups such as the elderly and visually impaired, resulting in low ordering efficiency and a poor user experience when ordering through robots. Summary of the Invention

[0004] The purpose of this application is to provide a robot-based beverage ordering method, apparatus, and electronic device to overcome or at least partially solve the above-mentioned problems.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows: A robot-based beverage ordering method, the method comprising: The system acquires the user's multimodal interaction information and user information; the multimodal interaction information and the user information each have corresponding weight coefficients. Obtain the user's ordering environment information; The weighting coefficients of the multimodal interaction information and the user information are adjusted based on the order environment information. Recommended beverage information is determined based on the multimodal interaction information after adjusting the weighting coefficients and the user information; The recommended beverage information is presented to the user so that the user can order beverages based on the recommended beverage information.

[0006] In one embodiment of this application, the multimodal interaction information includes at least voice information, gestures, and touch commands; the robot is equipped with multimodal sensors, which include at least a microphone, a touchscreen, and a camera; acquiring the user's multimodal interaction information and user information includes: The user's voice information is collected through the microphone; wherein, the voice information is obtained by the robot engaging in at least one round of dialogue with the user; The user's touch commands are collected through the touchscreen; The camera captures the user's image, and the user image is input into a preset deep learning model to obtain the user's gestures and facial expressions output by the preset deep learning model; wherein, the multimodal interaction information and the user information are remotely sent by the application.

[0007] In one embodiment of this application, the ordering environment information includes at least ambient noise, and adjusting the weighting coefficients of the multimodal interaction information and the user information based on the ordering environment information includes: When the ambient noise exceeds a preset ambient noise threshold, the weighting coefficient of the voice information is reduced, and the weighting coefficients of the gesture, the touch command, and the user information are increased.

[0008] In one embodiment of this application, the user information includes at least facial expressions and historical consumption records; the step of determining recommended beverage information based on the multimodal interaction information after adjusting the weighting coefficients and the user information includes: Obtain the multimodal perception information corresponding to the user; When the user is determined to be a returning customer based on the multimodal perception information, the user's preferred beverage information is determined based on the user's historical consumption records; wherein, the historical consumption records include at least the beverage information of the user's historical orders; the beverage information includes at least the beverage category, flavor, size, temperature, and sweetness. Target preferred beverage information is determined from the preferred beverage information based on the facial expressions after adjusting the weighting coefficients; the facial expressions reflect the user's emotional state. Recommended beverage information is determined from the target preferred beverage information based on the multimodal interaction information after adjusting the weighting coefficients; When the user is determined to be a new customer based on the multimodal perception information, a complete menu is displayed to the user for intelligent guidance.

[0009] In one embodiment of this application, the robot synchronizes with the POS system in real time through a preset standard interface. After determining the recommended beverage information based on the multimodal interaction information after adjusting the weighting coefficients and the user information, the method further includes: In response to the user's operation on the recommended beverage information, the target beverage information is determined from the recommended beverage information, and payment order information is generated for the target beverage information; In response to the user's operation on the payment order information, the payment method for the target beverage is provided to the user through the POS system; In response to the user's operation of the payment method, the user pays for the target beverage information through the POS system; the robot is used to prepare the beverage according to the target beverage information when the user completes the payment, and then deliver the beverage to the user after preparation.

[0010] In one embodiment of this application, determining the target preferred beverage information from the preferred beverage information based on the facial expression includes: When the emotional state is determined to be fatigued based on the facial expression, the preferred beverage information with a caffeine content higher than a first preset threshold is selected as the target preferred beverage information from the preferred beverage information. When the emotional state is determined to be pleasant based on the facial expression, the preferred beverage information is selected from the preferred beverage information, and the preferred beverage information with a sugar content higher than the second preset threshold or belonging to the current seasonal limited category is selected as the target preferred beverage information. When determining the emotional state as something other than fatigue or pleasure based on the facial expressions, the preferred beverage information of the classic or best-selling beverage in the preferred beverage information is taken as the target preferred beverage information.

[0011] In one embodiment of this application, determining recommended beverage information from the target preferred beverage information based on the multimodal interaction information after adjusting the weighting coefficients includes: Candidate beverage information is determined based on the multimodal interaction information after adjusting the weighting coefficients; When there is a conflict between the candidate beverage information and the target preferred beverage information, a first confidence level of the candidate beverage information and a second confidence level of the target preferred beverage information are obtained; the first confidence level reflects the credibility of determining the candidate beverage information based on the multimodal interaction information; the second confidence level reflects the credibility of the target preferred beverage information determined based on facial expressions. When the first confidence level is higher than the second confidence level, the target preferred beverage information that conflicts with the candidate beverage information is removed, and recommended beverage information is formed based on the removed target preferred beverage information and the candidate beverage information; When the first confidence level is lower than or equal to the second confidence level, an inquiry request is sent to the user based on the target preferred beverage information to obtain new multimodal interaction information of the user; Recommended beverage information is determined from the target preferred beverage information based on the new multimodal interaction information.

[0012] A robot-based beverage ordering device, the device comprising: A multimodal interaction information acquisition module is used to acquire the user's multimodal interaction information and user information; the multimodal interaction information and the user information each have corresponding weight coefficients; The order environment information acquisition module is used to acquire the user's order environment information; The weight coefficient adjustment module is used to adjust the weight coefficients of the multimodal interaction information and the user information according to the order environment information; The recommended beverage information determination module is used to determine recommended beverage information based on the multimodal interaction information after adjusting the weighting coefficients and the user information; The recommended beverage information module is used to recommend beverage information to the user so that the user can order beverages based on the recommended beverage information.

[0013] A robot for performing the above-described robot-based beverage ordering method, or including the above-described robot-based beverage ordering device.

[0014] An electronic device includes: a processor; and a memory for storing processor-executable instructions. The processor is configured to execute the instructions to implement the robot-based beverage ordering method described above.

[0015] A computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the above-described robot-based beverage ordering method.

[0016] The embodiments of this application have at least the following beneficial effects: In this embodiment, multimodal interaction information and user information are acquired. Each multimodal interaction information has a corresponding weight coefficient. The user's ordering environment information is also acquired to adjust the weight coefficients of the multimodal interaction information. Subsequently, recommended beverage information is determined and recommended to the user based on the adjusted multimodal interaction information and the user's information, allowing the user to place an order. The robot in this embodiment interacts with the user through multimodal methods such as voice, gestures, and touch to acquire the user's multimodal interaction information and user information. Furthermore, it can adaptively adjust the weight coefficients of the multimodal interaction information based on the current ordering environment information to avoid the ordering environment information affecting the accuracy of understanding the user's intent. Combining the adjusted multimodal interaction information and user information, the robot accurately understands the user's intent and determines and recommends beverage information that meets the user's needs, improving ordering efficiency and thus enhancing the user's robot-based ordering experience. In addition, since it allows users to interact with the user through multimodal methods to place orders, it is very user-friendly for special user groups (the elderly, visually impaired individuals). Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the steps of a robot-based beverage ordering method provided in this application embodiment; Figure 2 This is a system workflow diagram of a robot provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of a robot-based beverage ordering device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] The embodiments of this application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and are not intended to limit the scope of protection of this application.

[0019] Reference Figure 1 The diagram illustrates a flowchart of a robot-based beverage ordering method provided in this embodiment, which includes the following steps: Step 101: Obtain the user's multimodal interaction information and user information; the multimodal interaction information and the user information each have corresponding weight coefficients.

[0020] In specific implementations, the robot (robot control system) can be a humanoid robot, or it can be other types of robots capable of anthropomorphic interactive services; this application embodiment does not impose any limitations on this. The robot can interact with users through multimodal methods such as voice, gestures, touch, and QR code scanning, thereby obtaining multimodal interaction information such as the user's voice information, gesture actions, and touch commands to help the user place an order.

[0021] In this embodiment, the multimodal interaction information has a corresponding weight coefficient, which reflects the degree of credibility of understanding the user's intent when the user orders a beverage. This ensures that even if the quality of some multimodal data deteriorates, the robot can still accurately understand the actual user intent, thereby improving the user's ordering efficiency.

[0022] Initially, the weight coefficients for each multimodal interaction can be set to be the same. In addition to multimodal interaction information, the robot can also acquire user information, which may include at least facial expressions and historical consumption records.

[0023] Step 102: Obtain the user's ordering environment information.

[0024] Step 103: Adjust the weight coefficients of the multimodal interaction information and the user information according to the order environment information.

[0025] In this embodiment, when acquiring multimodal interaction information, the robot can obtain order environment information corresponding to the ordering environment in which the user places an order through the robot. Since the user's multimodal interaction information is obtained when the user is in the ordering environment, the multimodal interaction information is more or less affected by the ordering environment. Therefore, by adaptively adjusting the weight coefficients of each multimodal interaction information through order environment information, the accuracy of multimodal interaction information in understanding user intent can be improved.

[0026] In some embodiments, the ordering environment information may include at least ambient noise. Of course, the ordering environment information may also include the distance between the user and the robot, light intensity, etc.

[0027] Step 104: Determine recommended beverage information based on the multimodal interaction information after adjusting the weighting coefficients and the user information.

[0028] In this embodiment of the application, after adjusting the weighting coefficients, the robot can determine the recommended beverage information based on the multimodal interaction information and user information after adjusting the weighting coefficients. The recommended beverage information (beverage information) may include at least the beverage category, flavor, specifications, temperature, and sweetness.

[0029] Step 105: The recommended beverage information is presented to the user so that the user can order beverages based on the recommended beverage information.

[0030] In this embodiment, the robot can recommend beverage information to the user through multimodal methods such as screen, voice, or gesture. If the user determines that the recommended beverage information meets their needs, they can place an order based on the recommended beverage information. The robot can generate order information based on the order. After the user successfully pays according to the order information, the robot will prepare the corresponding beverage according to the order information and deliver the beverage through the robot's dispensing spout after it is prepared. At the same time, the robot can also guide the user to pick up the food through multimodal methods such as screen, voice, or gesture.

[0031] In this embodiment, multimodal interaction information and user information are acquired. Each multimodal interaction information has a corresponding weight coefficient. The user's ordering environment information is also acquired to adjust the weight coefficients of the multimodal interaction information. Subsequently, recommended beverage information is determined and recommended to the user based on the adjusted multimodal interaction information and the user's information, allowing the user to place an order. The robot in this embodiment interacts with the user through multimodal methods such as voice, gestures, and touch to acquire the user's multimodal interaction information and user information. Furthermore, the weight coefficients of the multimodal interaction information and user information can be adaptively adjusted based on the current ordering environment information to avoid the ordering environment information affecting the accuracy of understanding the user's intent. Combining the adjusted multimodal interaction information and user information, the robot accurately understands the user's intent and determines and recommends beverage information that meets the user's needs, improving the user's ordering efficiency and thus enhancing the user's robot-based ordering experience. In addition, since it allows users to interact with the user through multimodal methods to place orders, it is very user-friendly for special user groups (the elderly, visually impaired individuals, etc.).

[0032] In one embodiment of this application, the multimodal interaction information may include at least voice information, gestures, and touch commands; the robot may be equipped with multimodal sensors, which may include at least a microphone, a touchscreen, and a camera; acquiring the user's multimodal interaction information and user information may include: The user's voice information is collected through the microphone; wherein, the voice information is obtained by the robot engaging in at least one round of dialogue with the user; The user's touch commands are collected through the touchscreen; The camera captures the user's image, and the user image is input into a preset deep learning model to obtain the user's gestures and facial expressions output by the preset deep learning model; wherein, the multimodal interaction information and the user information are remotely sent by the application.

[0033] In practical implementation, the robot is equipped with multimodal sensors, including microphones, touchscreens, and cameras, which can collect multimodal interaction information from the user. The robot may include multiple modules such as a speech recognition module, a gesture recognition module, a facial expression analysis module, and a multimodal fusion engine.

[0034] The speech recognition module uses a far-field microphone array (8 microphones, beamforming technology) to collect the user's speech information, supporting multilingual recognition including Mandarin, Cantonese, and English, with a recognition rate of no less than 90% in a 60dB noise environment. In some embodiments, the speech information is obtained through at least one round of dialogue between the robot and the user. If the robot cannot understand the user's intention through one round of dialogue, or if the understood intention is incomplete, contradictory, or unreasonable, multiple rounds of dialogue can be triggered to guide the user to accurately express their intention through speech information. For example, the speech recognition module can have contextual understanding capabilities. Suppose the user mentions in the first round of dialogue that they do not want to order according to the default beverage information, the robot can further dialogue with the user to determine which specific default beverage information needs to be modified. For example, through multiple rounds of dialogue, the user can express that they want more sugar.

[0035] The gesture recognition module uses a depth camera (TOF (Time of Flight) technology, with a frame rate of 30fps (Frames Per Second)) to capture user images (hand images) for gesture recognition. It can recognize 12 predefined gesture actions (such as thumbs up to confirm, hand shake to cancel, pointing to select products, etc.), and the recognition distance is within a preset range.

[0036] The facial expression analysis module uses a high-resolution RGB camera (1080P resolution (progressive scan)) to capture user images (facial images), and then uses a deep learning model to identify various emotions (such as fatigue, joy, hesitation, satisfaction, dissatisfaction, neutrality, etc.). The recognition accuracy is high and is not lower than the preset accuracy threshold.

[0037] The multimodal fusion engine is the core innovative module of the robot, responsible for dynamically adjusting the weight coefficients of various multimodal interaction information and user information based on the ordering environment information.

[0038] The robot may also include a screen, through which users can perform touch operations such as clicking, swiping, and long-pressing to select beverage categories, specifications, and make payments. The robot will generate corresponding touch commands based on the touch operations.

[0039] It should be noted that the multimodal interaction information and user information can be sent to the robot remotely by the user through an application, or they can be sent to the robot in the store through screen touch, voice, or other means. This embodiment of the invention does not impose any restrictions on this.

[0040] This application embodiment obtains multimodal interaction information and user information from users through a robot. Compared with single interaction information, the robot can comprehensively and accurately understand the user's intention based on multimodal interaction information and user information, thereby improving the user's ordering efficiency and enhancing the user experience.

[0041] In one embodiment of this application, the ordering environment information includes at least ambient noise, and adjusting the weighting coefficients of the multimodal interaction information and the user information based on the ordering environment information may include: When the ambient noise exceeds a preset ambient noise threshold, the weighting coefficient of the voice information is reduced, and the weighting coefficients of the gesture, the touch command, and the user information are increased.

[0042] In this embodiment, the robot's multimodal fusion engine can dynamically adjust the weight coefficients of each multimodal interaction information based on the ordering environment information. In some embodiments, the ordering environment information may include at least environmental noise. When the environmental noise exceeds a preset environmental noise threshold (e.g., 65dB), the weight coefficient of voice information is automatically reduced, while the weight coefficients of gestures, touch commands, and user information (facial expressions) are increased. Then, the recommended beverage information is determined based on the multimodal interaction information and user information after the weight coefficients are adjusted, ensuring that the overall recognition accuracy is stable.

[0043] In one embodiment of this application, the user information includes at least facial expressions and historical consumption records; the step of determining recommended beverage information based on the multimodal interaction information after adjusting the weighting coefficients and the user information may include: Obtain the multimodal perception information corresponding to the user; When the user is determined to be a returning customer based on the multimodal perception information, the user's preferred beverage information is determined based on the user's historical consumption records; wherein, the historical consumption records include at least the beverage information of the user's historical orders; the beverage information includes at least the beverage category, flavor, size, temperature, and sweetness. Target preferred beverage information is determined from the preferred beverage information based on the facial expressions after adjusting the weighting coefficients; the facial expressions reflect the user's emotional state. Recommended beverage information is determined from the target preferred beverage information based on the multimodal interaction information after adjusting the weighting coefficients; When the user is determined to be a new customer based on the multimodal perception information, a complete menu is displayed to the user for intelligent guidance.

[0044] In practice, a user may be a new customer or a returning customer. Multimodal perception information can be obtained to determine whether a customer is new or returning. This multimodal perception information can include unique identifiers such as the user's voiceprint, fingerprint, or facial image.

[0045] For customers who have previously ordered food at the robot's stores (or chain stores), i.e., regular customers, the user information obtained, in addition to facial expressions, can include historical consumption records (user preferences). Based on historical consumption records, at least the user's previous beverage order information can be included. Beverage information includes at least the beverage category, flavor, size, temperature, and sweetness. Based on the beverage order information, the user's ordering preferences can be determined. Therefore, the user's preferred beverage information can be initially determined based on the beverage order information.

[0046] After determining the user's preferred beverage information, the robot can further determine the target preferred beverage information based on the user's emotional state as represented by facial expressions. Then, based on the multimodal interaction information after adjusting the weight coefficients, the robot determines the recommended beverage information from the target preferred beverage information, so that the recommended beverage information given to the user matches the user's current emotional state and ordering environment, thereby improving the user's ordering efficiency.

[0047] When a customer is new, a full menu can be displayed for intelligent guidance. Specifically, since stores lack information on new customers' preferences, displaying the full menu and providing intelligent guidance helps customers quickly understand the beverages available for order, thus facilitating a faster ordering process. Furthermore, after an order is placed, the customer's historical consumption records (customer preferences) can be recorded. This customer can then be considered a returning customer, allowing for preliminary determination of their preferred beverages based on their past orders, further improving ordering efficiency.

[0048] In one embodiment of this application, the robot synchronizes with the POS system in real time through a preset standard interface. After determining the recommended beverage information based on the multimodal interaction information after adjusting the weighting coefficients and the user information, the method may further include: In response to the user's operation on the recommended beverage information, the target beverage information is determined from the recommended beverage information, and payment order information is generated for the target beverage information; In response to the user's operation on the payment order information, the POS system provides the user with a payment method for the target beverage information; In response to the user's operation of the payment method, the user pays for the target beverage information through the POS system; the robot is used to prepare the beverage according to the target beverage information when the user completes the payment, and then deliver the beverage to the user after preparation.

[0049] In this embodiment, the robot connects to and synchronizes with the certified POS system in real time through a preset standard API (Application Programming Interface). Furthermore, the robot supports multiple payment methods (such as QR code scanning, NFC (Near Field Communication), facial recognition, etc.), so that users can achieve contactless interaction throughout the entire process of ordering, paying, and picking up food without having to make contact with the robot.

[0050] Specifically, after the robot pushes recommended beverage information to the user, if the user interacts with the recommended beverage information via voice, gesture, or touch, the robot responds by identifying the target beverage from the recommended beverage information and generating payment order information for the target beverage. If the user continues to interact with the payment order information, the robot responds by providing the user with payment methods for the target beverage through the POS system, such as QR code scanning, NFC, or facial recognition. The user can choose a payment method, and the robot responds by paying for the target beverage through the POS system. If the POS system confirms the payment is complete, the robot can begin preparing the beverage according to the target beverage information and deliver it to the user through the dispensing spout after preparation.

[0051] In one embodiment of this application, determining the target preferred beverage information from the preferred beverage information based on the facial expression may include: When the emotional state is determined to be fatigued based on the facial expression, the preferred beverage information with a caffeine content higher than a first preset threshold is selected as the target preferred beverage information from the preferred beverage information. When the emotional state is determined to be pleasant based on the facial expression, the preferred beverage information is selected from the preferred beverage information, and the preferred beverage information with a sugar content higher than the second preset threshold or belonging to the current seasonal limited category is selected as the target preferred beverage information. When determining the emotional state as something other than fatigue or pleasure based on the facial expressions, the preferred beverage information of the classic or best-selling beverage in the preferred beverage information is taken as the target preferred beverage information.

[0052] In this embodiment of the application, the robot may further include a personalized recommendation engine, which is used to combine the user's facial expressions to filter out the preferred beverage information from the preferred beverage information with caffeine content higher than a first preset threshold as the target preferred beverage information to determine the target beverage information.

[0053] For example, when facial expressions indicate a user's emotional state as fatigued, high-caffeine beverages can be recommended. Therefore, beverages with caffeine content exceeding a first preset threshold can be used as target preferred beverage information. When facial expressions indicate a user's emotional state as pleasant, high-sugar or limited-edition beverages can be recommended. Beverages with sugar content exceeding a second preset threshold or belonging to the current season's limited-edition category can be used as target preferred beverage information. When facial expressions indicate a user's emotional state other than fatigue or pleasant, such as hesitation, satisfaction, or dissatisfaction, to avoid errors, the preferred beverage information of classic or best-selling beverages can be used as target preferred beverage information. Specifically, when the user is hesitant, the preferred beverage information of the top 3 best-selling beverages can be used as target preferred beverage information; when the user is satisfied, the preferred beverage information of the same series as the most historically repurchased preferred beverage in the current season can be used as target preferred beverage information; and when the user is dissatisfied, the preferred beverage information of classic beverages can be used as target preferred beverage information.

[0054] In one embodiment of this application, determining recommended beverage information from the target preferred beverage information based on the multimodal interaction information after adjusting the weighting coefficients may include: Candidate beverage information is determined based on the multimodal interaction information after adjusting the weighting coefficients; When there is a conflict between the candidate beverage information and the target preferred beverage information, a first confidence level of the candidate beverage information and a second confidence level of the target preferred beverage information are obtained; the first confidence level reflects the credibility of determining the candidate beverage information based on the multimodal interaction information; the second confidence level reflects the credibility of the target preferred beverage information determined based on facial expressions. When the first confidence level is higher than the second confidence level, the target preferred beverage information that conflicts with the candidate beverage information is removed, and recommended beverage information is formed based on the removed target preferred beverage information and the candidate beverage information; When the first confidence level is lower than or equal to the second confidence level, an inquiry request is sent to the user based on the target preferred beverage information to obtain new multimodal interaction information of the user; Recommended beverage information is determined from the target preferred beverage information based on the new multimodal interaction information.

[0055] In this embodiment, after the robot determines the target preferred beverage information based on user information (facial expressions and historical consumption records), it can determine the candidate beverage information based on the multimodal interaction information after adjusting the weight coefficients. Then, it combines the target preferred beverage information and the candidate beverage information to determine the recommended beverage information to be recommended to the user. Typically, the target preferred beverage information and the candidate beverage information can be deduplicated and then merged to obtain the recommended beverage information.

[0056] In some cases, there may be conflicts between the target preferred beverage information and the candidate beverage information. For example, the target preferred beverage information specifies a small cup, while the candidate beverage information specifies a large cup. In this case, we can obtain the first confidence level of the candidate beverage information and the second confidence level of the target preferred beverage information. The first confidence level reflects the credibility of the robot's determination of the candidate beverage information based on multimodal interaction information; the second confidence level reflects the credibility of the target preferred beverage information determined by the robot based on facial expressions.

[0057] When the first confidence level is higher than the second confidence level, it indicates that the credibility of the candidate beverage information is greater than that of the target preferred beverage information. Therefore, the target preferred beverage information that conflicts with the candidate beverage information can be removed, and recommended beverage information can be formed based on the removed target preferred beverage information and the candidate beverage information. When the first confidence level is lower than or equal to the second confidence level, it indicates that the credibility of the candidate beverage information is lower than or equal to that of the target preferred beverage information. In practical applications, ordering should prioritize the user's current multimodal interaction information. This suggests that the currently determined user intent may not be complete or accurate enough. Therefore, an inquiry request can be sent to the user based on the target preferred beverage information to obtain new multimodal interaction information. Recommended beverage information can then be determined based on this new multimodal interaction information and the target preferred beverage information. Specifically, new candidate beverage information can be re-determined based on the new multimodal interaction information until there is no conflict between the target preferred beverage information and the new candidate beverage information.

[0058] To enable those skilled in the art to better understand the embodiments of this application, specific examples are used for illustration below. (Refer to...) Figure 2 This is a system workflow diagram of a robot provided in an embodiment of this application. Specific steps may include: A customer (user) enters the robot's interaction area; multimodal perception is activated (facial recognition + wake word detection); multimodal perception determines whether the customer is a new or returning customer; if a new customer, a full menu can be displayed for intelligent guidance; if a returning customer, historical preferences (historical consumption records) can be accessed for personalized recommendations; multimodal ordering interaction is used, including scanning a code / app (for remote pre-ordering); gestures / touchscreen (for graphical selection); and speech recognition (natural language understanding). This information is then combined to perform intent understanding and order parsing; if the customer does not confirm the parsed order, the robot can engage in multiple rounds of dialogue with the user to confirm details before returning to perform intent understanding and order parsing. If the customer confirms the parsed order, it can be confirmed as complete and proceed to payment processing. Payment processing supports multiple payment methods. If the payment result is successful, the order is sent to the robot's production system to prepare the beverage, and the system waits for notification of the time and real-time progress. After the beverage is prepared, the user can be notified to pick it up, thus completing the user's order.

[0059] The robot in this application provides a more natural and convenient ordering experience by interacting with users in a multimodal manner (voice, gestures, touch, and QR code scanning). In particular, when ordering by voice, users can describe their needs in natural language (such as "a latte that's not too sweet"). The robot can collect the voice information through its microphone, and then perform natural language understanding on the voice information. The robot can automatically parse and generate corresponding beverage information and recommend it to the user, improving the efficiency and experience of ordering through the robot.

[0060] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of this application.

[0061] Reference Figure 3 The diagram illustrates a structural block diagram of a robot-based beverage ordering device provided in an embodiment of this application. The device may specifically include the following modules: The multimodal interaction information acquisition module 301 is used to acquire the user's multimodal interaction information and user information; the multimodal interaction information and the user information each have corresponding weight coefficients; The order environment information acquisition module 302 is used to acquire the user's order environment information; The weight coefficient adjustment module 303 is used to adjust the weight coefficients of the multimodal interaction information and the user information according to the order environment information; The recommended beverage information determination module 304 is used to determine recommended beverage information based on the multimodal interaction information after adjusting the weight coefficients and the user information; The recommended beverage information recommendation module 305 is used to recommend the recommended beverage information to the user so that the user can place an order based on the recommended beverage information.

[0062] In one embodiment of this application, the multimodal interaction information includes at least voice information, gestures, and touch commands; the robot is equipped with multimodal sensors, which include at least a microphone, a touchscreen, and a camera; the multimodal interaction information acquisition module 301 is used for: The user's voice information is collected through the microphone; wherein, the voice information is obtained by the robot engaging in at least one round of dialogue with the user; The user's touch commands are collected through the touchscreen; The camera captures the user's image, and the user image is input into a preset deep learning model to obtain the user's gestures and facial expressions output by the preset deep learning model; wherein, the multimodal interaction information and the user information are remotely sent by the application.

[0063] In one embodiment of this application, the ordering environment information includes at least ambient noise, and the weighting coefficient adjustment module 303 is used to include: When the ambient noise exceeds a preset ambient noise threshold, the weighting coefficient of the voice information is reduced, and the weighting coefficients of the gesture, the touch command, and the user information are increased.

[0064] In one embodiment of this application, the user information includes at least facial expressions and historical consumption records; the recommended beverage information determination module 304 is used for: Obtain the multimodal perception information corresponding to the user; When the user is determined to be a returning customer based on the multimodal perception information, the user's preferred beverage information is determined based on the user's historical consumption records; wherein, the historical consumption records include at least the beverage information of the user's historical orders; the beverage information includes at least the beverage category, flavor, size, temperature, and sweetness. Target preferred beverage information is determined from the preferred beverage information based on the facial expressions after adjusting the weighting coefficients; the facial expressions reflect the user's emotional state. Recommended beverage information is determined from the target preferred beverage information based on the multimodal interaction information after adjusting the weighting coefficients; When the user is determined to be a new customer based on the multimodal perception information, a complete menu is displayed to the user for intelligent guidance.

[0065] In one embodiment of this application, the apparatus further includes: a payment module, used for: In response to the user's operation on the recommended beverage information, the target beverage information is determined from the recommended beverage information, and payment order information is generated for the target beverage information; In response to the user's operation on the payment order information, the payment method for the target beverage is provided to the user through the POS system; In response to the user's operation of the payment method, the user pays for the target beverage information through the POS system; the robot is used to prepare the beverage according to the target beverage information when the user completes the payment, and then deliver the beverage to the user after preparation.

[0066] In one embodiment of this application, the recommended beverage information determination module 304 is used for: When the emotional state is determined to be fatigued based on the facial expression, the preferred beverage information with a caffeine content higher than a first preset threshold is selected as the target preferred beverage information from the preferred beverage information. When the emotional state is determined to be pleasant based on the facial expression, the preferred beverage information is selected from the preferred beverage information, and the preferred beverage information with a sugar content higher than the second preset threshold or belonging to the current seasonal limited category is selected as the target preferred beverage information. When determining the emotional state as something other than fatigue or pleasure based on the facial expressions, the preferred beverage information of the classic or best-selling beverage in the preferred beverage information is taken as the target preferred beverage information.

[0067] In one embodiment of this application, the recommended beverage information determination module 304 is used for: Candidate beverage information is determined based on the multimodal interaction information after adjusting the weighting coefficients; When there is a conflict between the candidate beverage information and the target preferred beverage information, a first confidence level of the candidate beverage information and a second confidence level of the target preferred beverage information are obtained; the first confidence level reflects the credibility of determining the candidate beverage information based on the multimodal interaction information; the second confidence level reflects the credibility of the target preferred beverage information determined based on facial expressions. When the first confidence level is higher than the second confidence level, the target preferred beverage information that conflicts with the candidate beverage information is removed, and recommended beverage information is formed based on the removed target preferred beverage information and the candidate beverage information; When the first confidence level is lower than or equal to the second confidence level, an inquiry request is sent to the user based on the target preferred beverage information to obtain new multimodal interaction information of the user; Recommended beverage information is determined from the target preferred beverage information based on the new multimodal interaction information.

[0068] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment. This application also provides an electronic device, such as... Figure 4 As shown, it includes a processor 1001, a device interface 1002, a memory 1003, and a bus 1004; Memory 1003 is used to store computer programs; The processor 1001 executes the above steps when executing the program stored in the memory 1003.

[0069] The bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0070] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0071] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0072] This application also provides a storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to execute the robot-based beverage ordering method of the foregoing embodiments.

[0073] This application also provides a robot for performing the above-described robot-based beverage ordering method, or includes the above-described robot-based beverage ordering device.

[0074] The algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other equipment. The structure required to construct such a device is obvious from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0075] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0076] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0077] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0078] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the sequencing device according to this application. This application can also be implemented as a device or apparatus program for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can take the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0079] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0080] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0081] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

[0082] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0083] It should be noted that the various data-related processes in the embodiments of this application are carried out in compliance with the relevant data protection laws and policies of the country where the location is located, and with the authorization granted by the owner of the corresponding device.

Claims

1. A robot-based beverage ordering method, characterized in that, The method includes: The system acquires the user's multimodal interaction information and user information; the multimodal interaction information and the user information each have corresponding weight coefficients. Obtain the user's ordering environment information; The weighting coefficients of the multimodal interaction information and the user information are adjusted based on the order environment information. Recommended beverage information is determined based on the multimodal interaction information after adjusting the weighting coefficients and the user information; The recommended beverage information is presented to the user so that the user can order beverages based on the recommended beverage information.

2. The method according to claim 1, characterized in that, The multimodal interaction information includes at least voice information, gestures, and touch commands. The robot is equipped with multimodal sensors, which include at least a microphone, a touch screen, and a camera. The acquisition of user multimodal interaction information and user information includes: The user's voice information is collected through the microphone; wherein, the voice information is obtained by the robot engaging in at least one round of dialogue with the user; The user's touch commands are collected through the touchscreen; The camera captures the user's image, and the user image is input into a preset deep learning model to obtain the user's gestures and facial expressions output by the preset deep learning model. The multimodal interaction information and the user information are sent remotely by the application.

3. The method according to claim 2, characterized in that, The ordering environment information includes at least ambient noise, and adjusting the weighting coefficients of the multimodal interaction information and the user information based on the ordering environment information includes: When the ambient noise exceeds a preset ambient noise threshold, the weighting coefficient of the voice information is reduced, and the weighting coefficients of the gesture, the touch command, and the user information are increased.

4. The method according to claim 2, characterized in that, The user information includes at least facial expressions and historical consumption records; the step of determining recommended beverage information based on the multimodal interaction information after adjusting the weighting coefficients and the user information includes: Obtain the multimodal perception information corresponding to the user; When the user is determined to be a returning customer based on the multimodal perception information, the user's preferred beverage information is determined based on the user's historical consumption records; wherein, the historical consumption records include at least the beverage information of the user's historical orders; the beverage information includes at least the beverage category, flavor, size, temperature, and sweetness. Target preferred beverage information is determined from the preferred beverage information based on the facial expressions after adjusting the weighting coefficients; the facial expressions reflect the user's emotional state. Recommended beverage information is determined from the target preferred beverage information based on the multimodal interaction information after adjusting the weighting coefficients; When the user is determined to be a new customer based on the multimodal perception information, a complete menu is displayed to the user for intelligent guidance.

5. The method according to claim 1, characterized in that, The robot synchronizes with the POS system in real time via a preset standard interface. After determining the recommended beverage information based on the multimodal interaction information after adjusting the weighting coefficients and the user information, the method further includes: In response to the user's operation on the recommended beverage information, the target beverage information is determined from the recommended beverage information, and payment order information is generated for the target beverage information; In response to the user's operation on the payment order information, the POS system provides the user with a payment method for the target beverage information; In response to the user's operation of the payment method, the user pays for the target beverage information through the POS system; the robot is used to prepare the beverage according to the target beverage information when the user completes the payment, and then deliver the beverage to the user after preparation.

6. The method according to claim 4, characterized in that, The step of determining the target preferred beverage information from the preferred beverage information based on the facial expression includes: When the emotional state is determined to be fatigued based on the facial expression, the preferred beverage information with a caffeine content higher than a first preset threshold is selected as the target preferred beverage information from the preferred beverage information. When the emotional state is determined to be pleasant based on the facial expression, the preferred beverage information is selected from the preferred beverage information, and the preferred beverage information with a sugar content higher than the second preset threshold or belonging to the current seasonal limited category is selected as the target preferred beverage information. When determining the emotional state as something other than fatigue or pleasure based on the facial expressions, the preferred beverage information of the classic or best-selling beverage in the preferred beverage information is taken as the target preferred beverage information.

7. The method according to claim 6, characterized in that, The step of determining recommended beverage information from the target preferred beverage information based on the multimodal interaction information after adjusting the weighting coefficients includes: Candidate beverage information is determined based on the multimodal interaction information after adjusting the weighting coefficients; When there is a conflict between the candidate beverage information and the target preferred beverage information, a first confidence level of the candidate beverage information and a second confidence level of the target preferred beverage information are obtained; the first confidence level reflects the credibility of determining the candidate beverage information based on the multimodal interaction information; the second confidence level reflects the credibility of the target preferred beverage information determined based on facial expressions. When the first confidence level is higher than the second confidence level, the target preferred beverage information that conflicts with the candidate beverage information is removed, and recommended beverage information is formed based on the removed target preferred beverage information and the candidate beverage information; When the first confidence level is lower than or equal to the second confidence level, an inquiry request is sent to the user based on the target preferred beverage information to obtain new multimodal interaction information of the user; Recommended beverage information is determined from the target preferred beverage information based on the new multimodal interaction information.

8. A robot-based beverage ordering device, characterized in that, The device includes: A multimodal interaction information acquisition module is used to acquire the user's multimodal interaction information and user information; the multimodal interaction information and the user information each have corresponding weight coefficients; The order environment information acquisition module is used to acquire the user's order environment information; The weight coefficient adjustment module is used to adjust the weight coefficients of the multimodal interaction information and the user information according to the order environment information; The recommended beverage information determination module is used to determine recommended beverage information based on the multimodal interaction information after adjusting the weighting coefficients and the user information; The recommended beverage information module is used to recommend beverage information to the user so that the user can order beverages based on the recommended beverage information.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to execute the instructions to implement the robot-based beverage ordering method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the robot-based beverage ordering method as described in any one of claims 1 to 7.

Images