Beverage recommending and making system and method based on user emotion recognition

Abstract

The invention relates to the field of intelligent beverage recommendation, and discloses a beverage recommendation and making system and method based on user emotion recognition, and the system comprises a camera collection module which is used for collecting and generating a video stream; the emotion recognition module is used for receiving the video stream and generating structured emotion data; the recommendation module is used for receiving the structured emotion data and generating a beverage recommendation decision; and the control module is used for receiving the beverage recommendation decision, confirming the final beverage selection of the user, and controlling the beverage machine to complete beverage making according to the final beverage selection. According to the beverage recommendation system, the user face image is acquired and analyzed in real time through the camera acquisition module and the emotion recognition module to recognize the current emotion category of the user, and beverage recommendation is generated based on the emotion category, so that the recommendation result can dynamically respond to the instant psychological demand of the user, and the personalized and intelligent level of beverage recommendation is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent beverage recommendation, specifically to a beverage recommendation and preparation system and method based on user emotion recognition. Background Technology

[0002] With the rapid development of artificial intelligence and service robot technologies, automated beverage equipment is becoming increasingly common. Existing automated beverage machines typically allow users to place orders via touchscreen, voice, or mobile devices, with robotic arms or internal modules automatically preparing the drinks. However, most current beverage machine systems still rely on users actively selecting drinks and lack the ability to provide personalized recommendations based on the user's current emotional state.

[0003] Emotional state significantly influences users' beverage choices. For example, when tired, users tend to choose energizing coffees; when in a good mood, refreshing drinks may be more suitable; and when stressed or anxious, soothing milk teas or low-alcohol cocktails are more appropriate. Current beverage machines lack the ability to automatically recognize user emotions and combine this with beverage recommendations, indicating room for improvement in user experience and overall intelligence.

[0004] Currently, some research and products attempt to recommend beverages based on users' historical consumption data. However, relying solely on historical records cannot respond in real time to fluctuations in beverage demand caused by users' current emotional changes, making it difficult to achieve dynamic and accurate recommendations. In addition, some studies have attempted to use physiological indicators (such as heart rate and skin conductance) to help determine the user's state, but these methods usually require users to wear additional devices, which are cumbersome to operate and have low user acceptance, making it difficult to achieve contactless and rapid emotion recognition and recommendations.

[0005] Lacking an emotion-driven recommendation mechanism, existing beverage machines still require users to manually browse menus and make selections when they don't have a specific beverage need or want to regulate their mood through drinks. This increases interaction time and operational complexity, reducing beverage acquisition efficiency. Furthermore, the inability of recommended content to accurately match the user's current emotional needs leads to decreased user satisfaction with the taste and experience of the beverages, thus affecting the frequency of use and market acceptance of automatic beverage machines.

[0006] In summary, existing technologies still have significant shortcomings in integrating emotion recognition and beverage recommendation. There is an urgent need for an intelligent system that can perceive user emotions without contact and realize real-time personalized beverage recommendations and preparation, so as to improve the naturalness of human-computer interaction and the matching accuracy of beverage recommendations. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a beverage recommendation and preparation system and method based on user emotion recognition. This solves the problems of existing technologies being unable to perceive user emotions in real time and without contact, resulting in low matching degree between beverage recommendations and users' current needs, low level of intelligence, and poor user interaction experience.

[0008] To achieve the above objectives, the first aspect of the present invention provides a beverage recommendation and preparation system based on user emotion recognition, the system comprising: The camera acquisition module is used to capture continuous image frames containing the user's face and encode them into a video stream; An emotion recognition module is used to receive the video stream, process the images to identify the user's emotion category, and generate structured emotion data. The recommendation module is used to receive the structured emotion data and generate beverage recommendation decisions based on a preset recommendation strategy. The control module is used to receive the beverage recommendation decision, interact with the user through a human-computer interaction interface to confirm the user's final beverage selection, and control the beverage machine to complete the beverage preparation based on the final beverage selection.

[0009] In one specific implementation, the emotion recognition module deploys an emotion classification model based on a convolutional neural network (CNN), consisting of a cascaded feature extraction part and a classification part. The feature extraction part extracts deep-level abstract features from the input facial image through multiple convolutional blocks, where a rectified linear unit (ReLU) can be used as the activation function, defined as: In the formula, The input values ​​are the values ​​for the neurons. After the classification part outputs the raw predicted scores for each emotion category, it is transformed into a probability distribution using a softmax function. For the _th ... The formula for calculating the posterior probability pk of each emotion category is: ; In the formula, It comes from the aforementioned fully connected output layer, corresponding to the first... The original predicted scores for each category; It represents the total number of emotion categories.

[0010] In one specific implementation, the recommendation module employs a hybrid recommendation strategy, comprising a rule-based recommendation submodule and a machine learning-based personalized recommendation submodule, and selects between the two through a dynamic switching mechanism. This dynamic switching mechanism's decision logic is based on two conditions: first, whether the total number of historical interactions associated with the current user's identity is lower than a preset minimum interaction threshold; and second, whether the recommendation acceptance rate of the machine learning-based personalized recommendation submodule within the most recent sliding time window is lower than a preset minimum acceptance rate threshold. When either condition is triggered, the rule-based recommendation submodule is selected; otherwise, the machine learning-based personalized recommendation submodule is selected.

[0011] In one specific implementation, the core of the rule-based recommendation submodule is a rule base. This rule base predefines the mapping relationship from emotion categories to a set of beverage target attribute constraints. For example, fatigue is mapped to the attribute constraint set {caffeine intensity: high, flavor type: bitter}. This submodule uses this constraint set to query and filter in the beverage database to generate recommendations.

[0012] In one specific implementation, the machine learning-based personalized recommendation submodule employs a Wide&Deep model. This model is capable of parallel processing to achieve memorization and generalization, and it combines four types of features: sentiment features, user history features, beverage-specific features, and contextual spatiotemporal features. The model's total output... , which is the user's preference score for candidate beverage i, is calculated using the following formula: ; In the formula, It is the model's prediction of the current user's choice of candidate beverages. The preference score, which ranges from 0 to 1, can be interpreted as the probability that a user will have a positive interaction with the beverage (e.g., purchase). It is the Sigmoid logic function, which maps the sum of the terms in the parentheses to the interval (0, 1), making it a probability value; This is the output of the Wide part of the model, where, It is a feature vector that includes the original sparse features and the cross features; It is the weight vector corresponding to the feature vector, which is learned during the model training process; This represents the inner product of the two vectors, which is the prediction result of the linear model; It is the output of the Deep part of the model, which is the final result obtained after taking the embedding vectors of various features as input and passing them through a series of nonlinear transformations of the deep neural network; It is the embedding vector representing the current user; It is a representative candidate beverage The embedding vector; It is an embedding vector representing the current user's emotion; It is an embedding vector representing the spatiotemporal information of the current context; : is a scalar bias term, one of the parameters learned by the model.

[0013] To enable continuous optimization of the Wide&Deep model, this system can also incorporate a closed-loop learning mechanism. This mechanism quantifies user terminal interaction behaviors (such as placing an order directly, abandoning a purchase, or placing an order after making a change) into data with positive samples (…). ) or negative samples ( The system uses training samples labeled ) to periodically train the model by minimizing a binary cross-entropy loss function L with L2 regularization, using these new samples. The loss function is defined as: ; In the formula, It is the total loss value on the entire training dataset, which is the objective that needs to be minimized during the training process; It is the total number of samples in the training dataset; It is the first The true label of each training sample, with a value of 0 or 1; The model is for the first The probability that a training sample is labeled 1 is predicted, which is the output score of the aforementioned Wide&Deep model. It is a hyperparameter, namely the regularization strength coefficient, which is used to control the contribution of the regularization term to the total loss; It is the square of the L2 norm of all trainable weights (excluding bias terms) in the model, that is, the sum of the squares of all weight values.

[0014] In one specific implementation, the control module deploys a beverage recipe database. This database is used to translate the user's final, application-oriented beverage selection into a hardware-oriented, ordered sequence of hardware instructions containing a series of precise parameters. For example, a sweetened latte is translated into a list of instructions containing parameters such as the specific grams of powder to be ground, the amount of water dispensed, the water temperature, the amount of milk, and the amount of syrup, and this instruction sequence is sent sequentially to the target actuators of the beverage machine.

[0015] In one specific implementation, the camera acquisition module may be equipped with a low-power standby and automatic wake-up activation mechanism. This mechanism uses an independent passive infrared sensor to detect human movement. When a user is detected entering the effective interaction area, a wake-up interrupt signal is generated to trigger the camera acquisition module to enter the full-function working state from the low-power state.

[0016] A second aspect of this invention provides a method for recommending and preparing beverages based on user emotion recognition, comprising the following steps: Capture user images: When a user is detected, capture consecutive image frames containing the user's face through the camera and output the encoded video stream; Extracting features and identifying emotions: Based on the video stream, the images are processed and identified, and structured emotion data containing emotion categories is output; Based on the structured emotion data, a beverage recommendation decision is output by matching beverages according to the preset recommendation strategy. Output recommendation results: Based on the beverage recommendation decision, the recommendation information is presented to the user through the display screen and speaker, awaiting user instructions; User confirmation or change to another beverage: Based on the user's command received from the touch input channel or voice input channel, the final beverage selection of the user is parsed and output; Drive the beverage machine to make beverages: Based on the final beverage selection, translate it into a hardware instruction sequence and send the hardware instruction sequence to the beverage machine; Beverage delivery to user: After the beverage machine completes the beverage preparation according to the hardware instruction sequence, it delivers the beverage to the user.

[0017] This invention provides a beverage recommendation and preparation system and method based on user emotion recognition. It has the following beneficial effects: 1. This invention, by setting up a camera acquisition module and an emotion recognition module, can acquire and analyze the user's facial images in real time to identify the user's current emotion category, and generate beverage recommendations based on the emotion category. Compared with the traditional recommendation mode that relies on a fixed menu or long-term purchase history, this invention can enable the recommendation results to dynamically respond to the user's immediate psychological needs, thereby improving the personalization and intelligence level of beverage recommendations.

[0018] 2. This invention integrates rule-based and machine learning-based recommendation sub-modules into the recommendation module and intelligently selects between them through a dynamic switching mechanism. This allows the invention to balance service capabilities for new users with personalized accuracy for existing users, ensuring the stability and reliability of the system when user data is insufficient or model performance is poor. Furthermore, the machine learning recommendation, combined with a closed-loop learning mechanism, can continuously learn from user feedback and optimize the model, giving the invention's recommendation capabilities adaptive evolutionary characteristics and guaranteeing long-term service quality and accuracy.

[0019] 3. This invention combines emotion recognition, midstream intelligent recommendation, and downstream automated beverage production to form a closed loop from perception to execution. Therefore, this invention can not only understand user emotions and provide recommendations, but also convert the user's final choice into a sequence of hardware instructions for the beverage machine through the control module, thus achieving fully automated production. This reduces the user's operational burden and improves the human-computer interaction experience. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the system architecture of the present invention; Figure 2 This is a schematic diagram of the method flow of the present invention; Detailed Implementation The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] To better understand the present invention, the above content will be described in detail below with reference to specific embodiments.

[0022] Please see the appendix Figure 1 This invention provides a beverage recommendation and preparation system based on user emotion recognition. The system includes: a camera acquisition module, an emotion recognition module, a recommendation module, and a control module. In a specific embodiment: The camera acquisition module includes an RGB high-definition camera. In a preferred embodiment, the resolution is not less than 1920×1080 (1080P) and the frame rate is not less than 25 frames per second (fps). This ensures that the acquired single-frame image can contain sufficiently rich facial texture and detail information, providing high-quality data for the subsequent emotion recognition module. The camera acquisition module can be fixedly installed above the interactive area of ​​the beverage machine, such as in the center of the top bezel of the display screen, and has a preset downward tilt angle so that the camera's field of view can fully cover a user whose height is within the normal adult range, and can acquire clear facial images within a distance range of 50 cm to 150 cm.

[0023] Furthermore, to adapt to varying lighting conditions in real-world environments, the camera can also integrate automatic image quality adjustment functions. These functions specifically include automatic exposure (AE), automatic white balance (AWB), and automatic focus (AF) for the face area. The automatic exposure function dynamically adjusts the sensor's exposure time and gain based on the ambient light intensity, ensuring that the facial area is not overexposed or underexposed under normal indoor lighting conditions with an illuminance range of 100 to 800 lux. The automatic white balance function corrects color shifts in the image caused by light sources of different color temperatures. The automatic focus function for the face area prioritizes the sharpness of the user's face, rather than the background. The implementation methods of these functions are well-known technologies in the field and will not be elaborated upon here.

[0024] Furthermore, the camera acquisition module can be configured with a low-power standby and automatic wake-up activation mechanism. When there is no user interaction, it is in standby mode, at which point the camera module can be in a power-off or low-power mode. Then, an independent, extremely low-power detection unit continuously monitors the interaction area. In one embodiment, this detection unit can be a passive infrared sensor used to detect changes in infrared radiation caused by human movement. Therefore, when a user is detected entering the effective interaction area, a wake-up interrupt signal is immediately generated. This signal triggers the main image sensor of the camera acquisition module to power on and enter full-function operation, beginning the aforementioned high-quality image acquisition.

[0025] Once in full-function mode, the camera acquisition module encodes the captured continuous image frames into a video stream of a specific format and transmits it to the emotion recognition module. Embodiments of this invention support multiple video stream output formats, such as MJPEG or YUV. When using MJPEG, each frame in the video stream is an independent, JPEG-compressed image, which reduces the decoding complexity at the receiving end (i.e., the emotion recognition module). When using YUV, the video stream transmits uncompressed or losslessly compressed raw color space data, thus preserving the most complete image information, but requiring higher transmission bandwidth.

[0026] Furthermore, to ensure that image data can be transmitted from the camera acquisition module to the emotion recognition module in real time, a high-speed communication interface can be used. In one embodiment, the interface can be a Universal Serial Bus (USB) 3.0 interface. In another embodiment, the interface can also be an Ethernet interface. Both interfaces can provide sufficient bandwidth to meet the requirements of 1080P resolution, 25fps video streaming, and ensure that the end-to-end transmission delay of any single frame from acquisition to reception by the emotion recognition module does not exceed 100 milliseconds.

[0027] After the camera acquisition module transmits the image data stream to the emotion recognition module through a high-speed communication interface, the emotion recognition module begins to process the image. It can transform the original images, which vary in size and pose, into data with a uniform format and regular features, suitable for input into the subsequent convolutional neural network for analysis.

[0028] The first step is face detection. For each frame received from the video stream, a face detection algorithm is called to locate the rectangular region containing the face in the image; this region is referred to as the bounding box in this paper. In a specific embodiment, face detection is implemented using a multi-task cascaded convolutional neural network (MTCNN). MTCNN uses its three cascaded networks (P-Net, R-Net, O-Net) to progressively filter and refine candidate face boxes from the image, ultimately outputting high-precision face bounding box coordinates. This bounding box is determined by the coordinates of its top-left and bottom-right vertices.

[0029] When multiple facial regions are detected in a single frame of a video stream, a selection strategy is needed to determine the unique target user. One specific implementation involves calculating the area of ​​each detected bounding box and selecting the face corresponding to the bounding box with the largest area as the target user.

[0030] After determining the bounding box of the target face, to eliminate the impact of head pose changes (such as rotation and pitch) on the accuracy of emotion recognition, face alignment can be performed. This is done by utilizing facial key points acquired simultaneously during the face detection phase, such as the centers of the eyes, the tip of the nose, and the corners of the mouth—five or more key points in total. The essence of face alignment is to calculate an affine transformation matrix that maps the positions of the detected facial key points to the positions of template key points in a predefined standard coordinate system.

[0031] Specifically, this affine transformation is performed through a 3×3 transformation matrix. To achieve this, it will take the coordinates of points in the source image ( (Mapped to point coordinates in the target image) This mapping relationship can be described by the following formula: ; In the formula, the matrix The six parameters: ( , , , , , The transformation matrix is ​​obtained by solving a least-squares problem based on multiple pairs of keypoints (detected keypoints and template keypoints). The method for calculating this transformation matrix is ​​well-known in the art.

[0032] After calculating the transformation matrix This transformation is then applied to the entire face region defined by the bounding box in the original image, generating a pose-corrected, aligned face image. The system then crops this aligned face image to ensure uniform size. In one embodiment, all face images are uniformly cropped and scaled to 224×224 pixels.

[0033] Finally, the cropped and aligned face image is normalized in terms of pixel values. The purpose of this step is to linearly or non-linearly map the image's pixel values ​​from their original integer range (e.g., 0-255) to a floating-point range (e.g., 0 to 1, or -1 to 1) that is more suitable for neural network processing. In a preferred embodiment, normalization employs a channel-based Z-score normalization method, calculated as follows: ; in, It is the normalized coordinates Pixel value at; These are the pixel values ​​of the original image at the same coordinates; and These are the mean and standard deviation of all pixels in the color channel (R, G, B) of the pixel in question. These mean and standard deviation can be pre-calculated statistical values ​​from a large dataset. After this step, the standardized image data can be used as input for subsequent sentiment classification models.

[0034] After image processing, the generated standardized image data is fed into the emotion classification model based on a convolutional neural network (CNN) within the emotion recognition module. To meet the performance requirements of real-time inference on embedded hardware, the model in this embodiment employs a lightweight network structure. The model consists of a cascaded feature extraction part and a classification part.

[0035] In one specific embodiment, the feature extraction part of the model consists of four convolutional blocks. Its input is a normalized image tensor of size 224×224×3, processed in the previous stage. Specifically: The first convolutional block contains a convolutional layer using 32 3×3 kernels with a stride of 1, employing a same padding strategy to maintain the spatial dimensions of the feature maps. Following the convolutional layer is a batch normalization layer to accelerate model convergence and improve stability. Subsequently, the data is passed through a non-linear activation function, in this embodiment, the Corrected Linear Unit (ReLU), mathematically defined as: ; In the formula, The input value for the neuron. This is the output value after activation. Finally, a 2×2 max pooling layer with a stride of 2 is used to downsample the feature map, halving its spatial dimensions.

[0036] The subsequent three convolutional blocks repeat a similar structure to the first convolutional block: a combination of convolutional layer-normalization layer-ReLU activation-max pooling layer. The difference lies in the number of convolutional kernels used in each convolutional block, increasing progressively to extract deeper, more abstract features. Specifically, the second convolutional block uses 64 3×3 kernels, the third uses 128 3×3 kernels, and the fourth uses 256 3×3 kernels. All convolutional layers use a stride of 1 and the same padding, and all max pooling layers use a 2×2 size and a stride of 2.

[0037] After processing by four convolutional blocks, the original 224×224 input image is transformed into a high-dimensional feature map of size 14×14×256. This feature map is then flattened into a one-dimensional feature vector with dimensions of 14×14×256=50176.

[0038] This one-dimensional feature vector is then fed into the model's classification section. The classification section consists of two fully connected layers. The first fully connected layer receives a 50176-dimensional input vector and linearly transforms it to a hidden layer containing 512 neurons. This hidden layer also uses ReLU as the activation function. To prevent overfitting during training, a Dropout layer is placed after this hidden layer, which randomly sets the output of a subset of neurons to zero with a probability of 0.5 during training.

[0039] Finally, the fully connected layer is the model's output layer. It receives the 512-dimensional vector from the Dropout layer and linearly transforms it into an 8-dimensional output vector. The dimension of this vector corresponds to the total number of emotion categories defined in this invention (e.g., happy, relaxed, tense, tired, angry, sad, calm, surprised). Each element in this 8-dimensional vector represents the model's original predicted score for the corresponding emotion category, which will be converted into a probability value in subsequent processing steps.

[0040] After the classification part of the convolutional neural network outputs an 8-dimensional raw predicted score vector, the emotion recognition module performs probability transformation and decision steps to obtain the final structured emotion data that can be used by downstream modules.

[0041] The 8-dimensional original predicted score vector is denoted as Each element of the vector corresponds to a non-normalized score for a predefined emotion category. To transform these scores into a distribution that conforms to probability axioms, this vector is fed into a Softmax function. The Softmax function exponentializes each element in the vector and divides it by the sum of the exponents of all elements, thereby calculating each emotion category. posterior probability The calculation process is defined by the following formula: ; In the formula, The model predicts that the input image belongs to the first... The probability of each emotion category; It comes from the aforementioned fully connected output layer, corresponding to the first... The original predicted scores for each category; This represents the total number of emotion categories. In one specific embodiment of the present invention, These eight categories include happiness, relaxation, tension, fatigue, anger, sadness, calmness, and surprise. The formula ensures that all calculated probability values ​​are accurate. All values ​​are between 0 and 1, and the sum of the probabilities of all categories is 1.

[0042] After completing the Softmax calculation, an 8-dimensional probability distribution vector is obtained. Each element of this vector represents the confidence level of the corresponding emotion category of the input image. To determine the unique, most probable emotion category from this probability distribution, a maximum index operation can be performed on the probability vector. This operation returns the index of the element in the vector that has the maximum value.

[0043] The final identified emotion category Determined through the following decision function: ; in, This is the index corresponding to the predicted sentiment category, and along with the prediction result, its corresponding confidence score is also output, i.e., the index in the probability vector. value .

[0044] Ultimately, the identified emotion categories (Represented as a string) and its confidence score The data (represented as a floating-point number), along with a timestamp recording the time of identification, is encapsulated into a structured data object for transmission to the subsequent recommendation module via the internal network interface. In a preferred embodiment, this data object is in JSON format. A typical output example is as follows: {"emotion":"tired","confidence":0.91,"timestamp":"2025-10-09T11:30:00"}.

[0045] Then, the recommendation module receives standardized JSON-formatted sentiment data from the sentiment recognition module. To balance service capabilities for new users with personalized recommendation accuracy for existing users, a hybrid recommendation strategy is introduced in the recommendation module. This strategy includes two parallel recommendation paths: It includes a rule-based recommendation submodule and a machine learning-based personalized recommendation submodule, and through a dynamic switching mechanism, it automatically selects one of the paths to execute each time a recommendation needs to be generated, based on real-time conditions.

[0046] Specifically, the decision-making logic of this dynamic switching mechanism is mainly based on two dimensions of judgment conditions: The first condition concerns the completeness of historical user interaction data, used to handle the cold start problem for new or low-frequency users. First, the system queries the total number of historical interactions associated with the current user's identity, denoted as... Then, a minimum number of interactions is preset within the system. (For example, Before generating recommendations, the system first makes a judgment; if... If the current user data is insufficient to support effective inference by the personalized model, the switching mechanism will route the current recommendation task to the rule-based recommendation submodule.

[0047] The second condition concerns the online service performance of the machine learning-based personalized recommendation submodule, ensuring the stability and reliability of recommendation quality. Specifically, this involves continuously monitoring and calculating the recommendation acceptance rate of the machine learning model over a recent period. In a specific embodiment, this recommendation acceptance rate... Through a size of The sliding time window is used for calculation, for example, to statistically analyze the most recent... The number of times a recommendation generated by a machine learning model is ultimately accepted by the user (i.e., the user places an order directly or makes no changes) is divided by the total number of recommendations. The system also internally presets a minimum acceptance rate threshold for performance. (For example, ).

[0048] When satisfied The switching mechanism will further assess a second condition when sufficient user data is available. If the calculated recent recommendation acceptance rate... This indicates a decline or instability in the online performance of the machine learning model. To avoid providing users with a poor recommendation experience, the system will trigger a safety rollback logic, routing the current recommendation task to a more robust, rule-based recommendation submodule.

[0049] Only when neither of the above two conditions is triggered, i.e., the number of the user's historical interactions... And the recent recommendation acceptance rate of machine learning models Only with a dynamic switching mechanism will the default optimal path be selected, meaning that the current recommendation task will be handed over to the machine learning-based personalized recommendation submodule for execution.

[0050] To record and trace the strategy path used in each recommendation, the recommendation module includes a "source" field in the final output. This field explicitly identifies the submodule that generated the recommendation; for example, the value is "rule" when the recommendation is generated by the rule submodule, and "model" when it is generated by the machine learning submodule.

[0051] Furthermore, when the dynamic switching mechanism routes the recommendation task to the rule-based recommendation submodule, this submodule is activated to generate stable and logically consistent beverage recommendations. This rule-based recommendation submodule does not rely on any historical user data; its core is a pre-configured, static rule base.

[0052] In one implementation, this rule base can be a lookup table or a set of key-value pairs stored in the system's non-volatile memory. This data structure uses an emotion category string (e.g., tired) received from the emotion recognition module as the key for the query, and a set of corresponding target attribute constraints describing the beverage as the value. This associates emotions with specific beverages and their intrinsic attributes, thereby improving the system's flexibility and scalability.

[0053] In one specific embodiment, the rule base can also define a mapping relationship from mood categories to beverage attribute constraints, including but not limited to caffeine intensity (e.g., none, low, medium, high), temperature (e.g., iced, room temperature, warm, hot), and flavor type (e.g., bitter, sweet, sour, creamy, fruity). An exemplary subset of this rule base is as follows: If the input emotion is fatigue, the corresponding attribute constraints are: {caffeine intensity: high, flavor type: bitter}.

[0054] If the input emotion is tension, the corresponding attribute constraints are: {Caffeine intensity: none or low, temperature: warm or hot, flavor type: creamy or sweet}.

[0055] If the input emotion is "happy", the corresponding attribute constraint is: {Temperature: Ice, Flavor type: Fruity or Sweet}.

[0056] Upon receiving the emotion category input, this submodule first searches for the corresponding set of attribute constraints in the rule base. Then, using these attribute constraints, it queries and filters a separate beverage database containing all beverages that can be made in the system and their detailed attributes. For example, after obtaining the attribute constraints {caffeine intensity: high, flavor type: bitter} for the emotion of fatigue, this submodule will traverse the beverage database and filter out all beverages that simultaneously meet both conditions, such as Americano and espresso.

[0057] If multiple candidate beverages that meet the criteria are obtained after filtering, a preset sorting or selection strategy is used to determine the final recommendation. In one embodiment, this strategy could be a simple default priority list; for example, Americano is always recommended first over espresso. In another embodiment, the strategy could be to randomly select a candidate beverage to increase the diversity of recommendations.

[0058] The final selected beverage, along with the reason for recommendation (i.e., the emotion that triggered the rule) and the field identifying its source as "rule" as described in the aforementioned switching mechanism, are then encapsulated into a structured data object. This object is then sent to the control module for subsequent user interaction and presentation.

[0059] When the dynamic switching mechanism ultimately chooses to delegate the recommendation task to the machine learning-based personalized recommendation submodule based on factors such as the completeness of user historical data and the performance of the model's online service, the first step of this personalized recommendation submodule is to extract, process, and integrate information from multiple data sources to construct a structured, information-rich input feature vector for the subsequent Wide&Deep model. Specifically, this includes: Emotional features: These are directly derived from the JSON data object output by the upstream emotion recognition module. Specifically, the emotion category string (e.g., tired) is extracted and quantified before being fed into the model. A preferred implementation is to process it using one-hot encoding, which converts it into an 8-dimensional binary vector, where the dimension corresponding to the currently identified emotion category is 1, and the other 7 dimensions are 0. User historical features: These are derived from the user's historical interaction database stored in the system and are used to describe the user's long-term preferences and recent behavior. In one embodiment, these features specifically include: the ID sequence of the N drinks the user has recently purchased (e.g., N=5); and the user's historical preference distribution for different drink attributes (such as sweetness, temperature, and caffeine strength), calculated by statistically weighting all of the user's historical orders. For categorical features like drink ID sequences, one-hot encoding or conversion into embedding vectors is used for processing. For numerical features like preference distributions, they are directly used as input to the model. Beverage-specific characteristics: These describe the attributes of each candidate beverage to be recommended, and the data comes from a beverage database. Specific characteristics include: beverage ID, caffeine strength level, temperature type, main flavor type, and price. Among these, ID, caffeine strength, temperature, and flavor type are all categorical features and are processed using one-heat encoding before being input into the model. Price, as a continuous numerical feature, requires normalization to eliminate dimensional differences. A preferred implementation method is to use a min-max scaling approach.

[0060] Contextual spatiotemporal features: These are used to capture environmental information when recommendation behavior occurs, and the data comes from the system's clock module. Specific features include the time period in which the current interaction occurs (e.g., divided into five time periods: morning, noon, afternoon, evening, and late night) and the day of the week (e.g., Monday to Sunday). Both of these features are categorical features and are processed using one-hot encoding to enable the model to learn the differences in user consumption patterns at different times.

[0061] Therefore, through the above steps, these four types of raw data from different sources and with different properties are uniformly processed and combined into feature vectors to describe which user might choose which beverage under what mood and time and space, and are then sent as the final input to the Wide and Deep parts of the subsequent Wide&Deep model for processing.

[0062] After extracting and processing four key features and combining them into a high-dimensional feature vector, this vector is input into the Wide&Deep model. This Wide&Deep model achieves accurate prediction of user preferences through parallel Wide and Deep components. Specifically: The Wide part of the model is a generalized linear model whose main function is to memorize specific combinations of features. The features utilized in this part specifically include categorical features defined in the previous feature engineering stage and processed with one-hot encoding. These features specifically cover: emotion category as an emotion feature; beverage ID sequence as a user history feature; beverage ID, caffeine strength level, temperature type, and flavor type as beverage characteristics; and the time period and day of the week as contextual spatiotemporal features. Simultaneously, to capture specific interactions between different features, the Wide part also cross-combines these numerically processed categorical features to generate new cross-features. An example of a cross-feature is performing a logical AND operation between the emotion feature "fatigue" and the contextual spatiotemporal feature "morning" to generate a new binary feature "morning and fatigued." By learning the weights of these original features and cross-features, the Wide part can directly capture feature combinations that have repeatedly occurred historically and are highly correlated with user behavior, thereby achieving the ability to memorize specific rules. This allows the Wide&Deep model to respond quickly and accurately to scenarios it has encountered.

[0063] The Deep part is a deep neural network (DNN), whose main function is to explore and generalize complex relationships between features. For the high-dimensional sparse categorical features generated in the aforementioned feature engineering (e.g., user ID, beverage ID, emotion category, etc.), directly inputting them into a DNN would be difficult to train due to the excessive number of parameters. Therefore, in the Deep part, these high-dimensional sparse features are first fed into an embedding layer. The embedding layer is essentially a trainable lookup table used to map each discrete feature ID to a low-dimensional, dense real-valued vector, i.e., an embedding vector. Through learning during training, this embedding layer can map semantically similar features (e.g., beverages with similar flavors) to nearby positions in the embedding space.

[0064] After the embedding layer, the embedding vectors of all features are concatenated into a single, dense vector. This vector is then sequentially fed into a feedforward neural network containing multiple hidden layers. Each hidden layer consists of several neurons and uses a rectified linear unit (ReLU) as the activation function to learn and represent the highly complex, non-linear interactions between features. This deep structure enables the discovery of hidden patterns in the data that are not directly captured by cross-features, thus providing the ability to generalize to previously unseen combinations of features.

[0065] Finally, the linear predictions from the Wide part and the nonlinear predictions from the Deep part are combined to calculate the user's preference for a specific beverage. Preference score The total output of the model is defined by the following formula: ; In the formula, It is the model's prediction of the current user's choice of candidate beverages. The preference score, which ranges from 0 to 1, can be interpreted as the probability that a user will have a positive interaction with the beverage (e.g., purchase). It is the Sigmoid logic function, which maps the sum of the terms in the parentheses to the interval (0, 1), making it a probability value; This is the output of the Wide part of the model, where, It is a feature vector that includes the original sparse features and the cross features; It is the weight vector corresponding to the feature vector, which is learned during the model training process; It represents and The inner product of the linear model is the prediction result. It is the output of the Deep part of the model, which is the final result obtained after taking the embedding vectors of various features as input and passing them through a series of nonlinear transformations of the deep neural network; It is the embedding vector representing the current user; It is a representative candidate beverage The embedding vector; It is an embedding vector representing the current user's emotion; It is an embedding vector representing the spatiotemporal information of the current context; : is a scalar bias term, one of the parameters learned by the model.

[0066] In order for the aforementioned Wide&Deep model to continuously optimize its recommendation performance by learning user behavior and form an effective closed-loop learning system, it is first necessary to quantify the user's terminal interaction behavior and transform it into labeled data that can be used for supervised learning by the model.

[0067] Once the recommendation module presents one or more recommended drinks to the user, the user's final consumption behavior can be obtained and mapped to training samples. The training samples consist of the complete feature vector that leads to the recommendation and a paired binary label.

[0068] If a user places an order directly, it means the user has accepted the system's recommendation of a specific beverage and completed the purchase. This interaction is defined as a successful recommendation. The system will record a complete feature vector that generated this recommendation, containing user characteristics, emotional characteristics, contextual spatiotemporal characteristics, and the characteristics of the accepted beverage itself. This vector will be assigned a positive sample label to the record, which can be represented as follows: .

[0069] If the user's behavior is to abandon the purchase—that is, after seeing the recommendation, the user does not take any action within the preset decision time or actively exits the interactive interface—then this interaction is considered a failed recommendation, indicating that the recommended beverage failed to arouse the user's interest. In this case, the system will record the feature vector that generated this recommendation, assign it a negative sample label, and represent it as... .

[0070] When a user's behavior involves changing their order before placing it—for example, the system recommends drink A, but the user selects and orders another drink B from the menu—this behavior is interpreted as two independent training signals. First, the user's rejection of drink A generates a negative sample: the feature vector related to drink A is recorded and assigned a negative sample label. Secondly, the user's final choice of beverage B is considered a strong positive feedback. Therefore, the system will use the same user characteristics, emotional characteristics, and contextual spatiotemporal features from this interaction, combined with the features of the beverage B actually chosen by the user, to construct a new feature vector, and assign a positive sample label to this newly generated training sample record. .

[0071] Therefore, through the above quantization and labeling process, every real-time interaction with the user can be transformed into training data with clear supervision signals, and this data is the basis of the dataset required for subsequent Wide&Deep model iterative training.

[0072] After quantifying user terminal interaction behaviors and converting them into training samples, this data is used to periodically drive the parameter optimization of the aforementioned Wide&Deep model. This iterative training process first requires defining a loss function and then minimizing its value using an effective optimization algorithm.

[0073] In a preferred embodiment of the present invention, the loss function used is a binary cross-entropy loss with L2 regularization. Since the model's task is to predict whether a user will accept a recommendation (…), ) or refuse to recommend ( The binary probability (or cross-entropy) is the difference between the model's predicted probability and the true label. To prevent overfitting during training (i.e., excessive memorization of training data leading to a loss of generalization ability to new data), an L2 regularization term is added to the loss function to penalize excessively large model weights. Therefore, the complete loss function L2 can be defined as: ; In the formula, It is the total loss value on the entire training dataset, which is the objective that needs to be minimized during the training process; It is the total number of samples in the training dataset; It is the first The true label of each training sample, with a value of 0 or 1; The model is for the first The probability that a training sample is labeled 1 is predicted, which is the output score of the aforementioned Wide&Deep model. It is a hyperparameter, namely the regularization strength coefficient, which is used to control the contribution of the regularization term to the total loss; It is the square of the L2 norm of all trainable weights (excluding bias terms) in the model, that is, the sum of the squares of all weight values.

[0074] In order to minimize this loss function Gradient descent-based parameter update methods can be used to adjust all trainable parameters of the model (including the weights of the wide part). The embedding vectors of the Deep part and the weights and biases of each layer of the network. In one specific embodiment, the optimizer used is the Adaptive Moment Estimation (Adam) optimizer, which combines the advantages of the momentum method and the RMSProp algorithm. This allows for the calculation of an adaptive learning rate for each parameter in the Wide & Deep model, thereby achieving efficient and robust convergence. The specific implementation of the Adam optimizer can be achieved by those skilled in the art based on publicly available information; it is a well-known technique in the field and will not be elaborated upon here.

[0075] Therefore, the aforementioned training data collection and model parameter update process forms an automated intelligent evolution loop. New user feedback data is continuously collected from online interactions and transformed into labeled training samples according to the aforementioned rules, then stored in a dedicated training database. At this point, a retraining process can be automatically triggered periodically (e.g., every early morning) or after a sufficient number of new samples have been collected (e.g., every 1000 new records). In one embodiment, this process is periodic full retraining, which involves training a new model from scratch using a complete dataset containing all historical and latest data. In another embodiment, to reduce computational resource consumption, the process can also be incremental updates, where the model is fine-tuned using only newly collected data based on existing model parameters. After training, the new model, validated by offline evaluation metrics, is automatically deployed online, replacing the old model version, thereby achieving continuous, adaptive optimization and evolution of recommendation capabilities.

[0076] In one embodiment, after the recommendation module completes its iterative training process and outputs an optimal beverage recommendation decision, this recommendation decision is transmitted to the system's control module. Specifically, the control module is used to transform the abstract recommendation result into a concrete human-computer interaction, and to accurately parse and issue the user's final choice into a series of hardware instructions that can be executed by the beverage machine.

[0077] The specific steps are as follows: First, the human-computer interaction management function is executed. After receiving a JSON data object from the recommendation module containing fields such as the beverage name and the reason for recommendation, the control module first parses the data. Then, it calls the graphical user interface (GUI) rendering function to generate a recommendation information interface on the system's display screen. The interface presents the appearance, name, and reason for recommendation of the recommended beverage to the user in the form of a combination of text and images (for example, detecting that you are currently fatigued, we recommend an Americano to refresh you). At the same time, the control module calls a text-to-speech (TTS) engine to synthesize the same recommendation information into natural speech, which is then broadcast through the speaker unit, thereby improving the user experience.

[0078] After presenting the recommended information, the control module enters a waiting state for command reception, simultaneously managing two parallel input channels: a touch input channel and a voice input channel. For the touch input channel, the control module continuously listens for events from the touchscreen driver. When a touch event is detected occurring in a preset screen area (e.g., confirming an order or viewing other buttons), the event is interpreted as a user's confirmation or change command. For the voice input channel, the control module activates an Automatic Speech Recognition (ASR) engine that continuously listens for ambient sounds and is configured to specifically recognize a set of preset keywords (e.g., "okay," "I want this," "change," "no"). Once the ASR engine recognizes one of the keywords and reaches a preset confidence threshold, the recognition result is considered the user's voice command. To ensure the uniqueness of the command, once either channel receives a valid user command, the control module immediately locks and ignores subsequent input from the other channel, thus ensuring explicit reception of the user's final decision.

[0079] Furthermore, upon receiving the user's final confirmed beverage selection (e.g., a latte with sugar), the control module initiates the parsing and distribution process for the preparation task. The first step is instruction translation. Specifically, the control module maintains an internal beverage recipe database, using the beverage and all possible modifications (such as adding sugar or removing ice) as keys, and structured instruction sequences containing a series of precise parameters as values. The control module translates the user-selected, application-oriented beverage name into a specific sequence of instructions for each actuator in the beverage machine. For example, a cup of sweetened latte would be translated into an ordered list containing the following steps: [{Actuator: robotic arm, Action: pick up cup, Parameter: medium cup}, {Actuator: coffee bean grinder, Action: grind, Parameter: 18 grams}, {Actuator: espresso pump, Action: dispensing, Parameter: {water volume: 30 ml, water temperature: 92℃}}, {Actuator: milk pump, Action: dispensing, Parameter: {milk volume: 150 ml, temperature: 65℃}}, {Actuator: syrup pump, Action: dispensing, Parameter: {volume: 10 ml}}].

[0080] Finally, after generating the aforementioned hardware instruction sequence, the control module begins executing these instructions sequentially, i.e., performing hardware communication. For each instruction in the sequence, the control module constructs and sends a corresponding communication message based on the type of industrial bus to which its target actuator is connected. In an embodiment using a CAN bus, the control module constructs a CAN message frame, where the arbitration ID field identifies the target actuator (e.g., a milk pump controller), and the data field contains specific instruction parameters (e.g., discharge rate and target temperature). In another embodiment using the Modbus-RTU protocol, the control module constructs a Modbus-RTU frame, which includes the slave address, function code (e.g., write multiple registers), register start address, and the data to be written.

[0081] The sending of instructions is not a one-way process. After each instruction is issued, the control module enters a waiting state to monitor the task's completion. Specifically, it waits for an acknowledgment or status message to be returned from the target executor's controller. If an acknowledgment message indicating successful task completion is received within a preset timeout period, the control module retrieves the next instruction from the sequence and continues execution. If an error message is received or a timeout occurs, the control module immediately initiates a fault handling procedure, such as stopping all ongoing actions and reporting the error information to the user on the display screen, thereby ensuring the accuracy and safety of the entire physical execution process.

[0082] Please see the appendix Figure 2 This invention also provides a method for recommending and preparing beverages based on user emotion recognition, which includes the following steps: Step 1: Acquire user image When the passive infrared sensor detects that a user is approaching and entering the effective interaction area, the camera (e.g., a 1080P high-definition camera) in low-power standby mode is awakened and begins to acquire continuous image frames containing the user's face. These image frames are then encoded into video streams in MJPEG or YUV format and output in real time through a high-speed interface.

[0083] Step 2: Extract features and identify emotions Upon receiving the video stream, preprocessing operations such as face detection, face alignment, and image normalization are sequentially performed on each frame to generate standardized face images with uniform size (e.g., 224×224 pixels) and standard pose. These standardized images are then input into a pre-trained, lightweight convolutional neural network (CNN) emotion classification model. The model performs deep feature extraction and classification on the images, ultimately outputting structured emotion data containing eight preset emotion categories, such as happy, tired, and tense, along with their corresponding confidence scores, through a Softmax layer. This data is encapsulated in JSON format.

[0084] Step 3: Match drinks to mood and make recommendations The pre-defined hybrid recommendation strategy is activated. First, a dynamic switching mechanism is used to determine whether the current user is a new user (historical interaction count is below a threshold) or if the personalized recommendation model has performed poorly recently (recommendation acceptance rate is below a threshold). In this case, the built-in rule base is queried based on the user's mood (e.g., fatigue) to match the corresponding beverage attribute constraints (e.g., caffeine intensity: high), and beverages that meet the conditions (e.g., Americano) are selected as the recommendation decision. Conversely, if there is sufficient user data and the model performs well, the user's current mood characteristics, historical user characteristics, beverage characteristics, and contextual spatiotemporal characteristics are combined to predict the user's preference score for each candidate beverage, and the beverage with the highest score is selected to form the final recommendation decision.

[0085] Step 4: Output Recommendation Results Multimodal human-computer interaction is implemented. On one hand, the information of the recommended beverage and the reasons for the recommendation are presented on the display screen in the form of pictures and text (for example, if we detect that you are currently tired, we recommend an Americano). On the other hand, the same content is synthesized into speech through a text-to-speech (TTS) engine and broadcast through a speaker. Then the system enters a state of waiting for user commands.

[0086] Step 5: User confirms or changes to another beverage Simultaneously, the system listens for user commands from both the touch input channel and the voice input channel. Users can confirm recommendations by tapping the confirmation button on the screen or by saying keywords such as "okay" or "I want this" to the Automatic Speech Recognition (ASR) engine. Users can also browse the menu and select other drinks by choosing to view other options or saying "change one." The system analyzes the first valid command received and uses it as the user's final drink selection.

[0087] Step Six: Drive the beverage making equipment to make the beverage. After obtaining the user's final beverage selection, the system queries the internal beverage recipe database and translates the selected beverage name (e.g., sweetened latte) into an ordered sequence of hardware instructions containing precise parameters. This sequence defines each preparation step in detail, such as: {Actuator: coffee bean grinder, Action: grinding, Parameter: 18 grams}, {Actuator: syrup pump, Action: dispensing, Parameter: {Quantity: 10 ml}}, etc. Subsequently, the control module sends the instructions sequentially to the corresponding actuators (such as grinder, water pump, robotic arm, etc.) in the beverage machine via industrial communication protocols such as CAN bus or Modbus.

[0088] Step 7: Deliver the beverage to the customer The beverage machine completes all preparation steps based on the received hardware instruction sequence. Once prepared, the beverage is delivered to the user via a cup dispenser or robotic arm, thus ending the process.

Claims

1. A drink recommendation and making system based on user emotion recognition, characterized by, The method comprises the following steps: a camera acquisition module for acquiring continuous image frames containing the user's face and encoding them into a video stream; an emotion recognition module for receiving the video stream, processing the images to identify the user's emotion category, and generating structured emotion data; a recommendation module for receiving the structured emotion data and generating a beverage recommendation decision based on a pre-set recommendation strategy; a control module for receiving the beverage recommendation decision, interacting with the user through the beverage machine to confirm the user's final beverage selection, and controlling the beverage machine to complete beverage production according to the final beverage selection. 2.The beverage recommendation and making system based on user emotion recognition of claim 1, wherein, The emotion recognition module includes an emotion classification model based on a convolutional neural network, which is composed of a cascaded feature extraction part and a classification part; the feature extraction part is used to extract deep features from the image, and the classification part is used to output emotion category prediction scores according to the deep features. 3.The beverage recommendation and making system based on user emotion recognition of claim 1, wherein, The recommendation module includes a rule-based recommendation submodule and a machine learning-based personalized recommendation submodule, and a dynamic switching mechanism for selecting between the rule-based recommendation submodule and the personalized recommendation submodule. 4.The beverage recommendation and making system based on user emotion recognition of claim 3, wherein, The dynamic switching mechanism includes: when it is detected that the total number of historical interactions associated with the current user identity is lower than a pre-set minimum interaction number threshold, or when the recommendation acceptance rate of the machine learning-based personalized recommendation submodule within a recent sliding time window is lower than a pre-set minimum acceptance rate threshold, the rule-based recommendation submodule is selected to perform the recommendation task; otherwise, the machine learning-based personalized recommendation submodule is selected to perform the recommendation task. 5.The beverage recommendation and making system based on user emotion recognition of claim 3, wherein, The rule-based recommendation submodule specifically includes: a rule base for mapping emotion categories to a set of target attribute constraints of beverages, and the rule-based recommendation submodule queries and filters in the beverage database by using the target attribute constraints to determine the final recommended beverage. 6.The beverage recommendation and making system based on user emotion recognition of claim 3, wherein, The machine learning-based personalized recommendation submodule includes a Wide&Deep model for combining emotion features, user historical features, beverage features, and context spatiotemporal features to predict user preference scores for each candidate beverage. 7.The beverage recommendation and making system based on user emotion recognition of claim 6, wherein, The machine learning-based personalized recommendation submodule also includes a closed-loop learning mechanism for quantifying the user's terminal interaction behavior and converting it into training samples with positive or negative sample labels, and periodically using the training samples to iteratively train the Wide&Deep model by minimizing a binary cross-entropy loss function with L2 regularization. 8.The beverage recommendation and making system based on user emotion recognition of claim 1, wherein, The control module also includes a beverage recipe database for translating the user's final confirmed beverage selection into an ordered hardware instruction sequence containing a series of parameters, and sequentially issuing the hardware instruction sequence to the beverage machine. 9.The beverage recommendation and making system based on user emotion recognition of claim 1, wherein, The camera acquisition module is provided with a low-power standby and automatic wake-up activation mechanism. The low-power standby and automatic wake-up activation mechanism is provided with a passive infrared sensor for detecting human movement. When a user enters an effective interaction area, a wake-up interrupt signal is generated to trigger the camera acquisition module to enter a working state.

10. A beverage recommendation and preparation method based on user emotion recognition, based on the beverage recommendation and preparation system based on user emotion recognition of any one of claims 1-9, characterized in that, The method comprises the following steps: Acquiring user images: When a user is detected, continuous image frames containing the user's face are acquired by a camera, and an encoded video stream is outputted; Extracting features and identifying emotions: Based on the video stream, images are processed and identified, and structured emotional data containing emotional categories are outputted; Matching drinks according to emotions and making recommendations: Based on the structured emotional data, a preset recommendation strategy is used for matching, and a drink recommendation decision is outputted; Outputting recommendation results: Based on the drink recommendation decision, recommendation information is presented to the user through a display screen and a loudspeaker, waiting for user instructions; User confirmation or replacement of other drinks: Based on the user instructions received from a touch input channel or a voice input channel, the user's final drink selection is analyzed and outputted; Driving the drink machine to make drinks: Based on the final drink selection, a hardware instruction sequence is translated, and the hardware instruction sequence is sent to the drink machine; Delivering drinks to users: After the drink machine completes the drink making according to the hardware instruction sequence, the drink is delivered to the user.

Images