Intelligent gesture-based ordering system

Abstract

This disclosure provides an intelligent gesture ordering system, which includes a camera device and an interactive device. The camera device captures and transmits the user's image; the interactive device plays an activation voice and receives the user's image generated according to the activation voice. The interactive device processes the user's image and displays various food and beverage items when it determines that the user's hand gesture matches the activation gesture. The user selects food and beverage items and their corresponding specifications by inputting their image through the camera device. The interactive device integrates the selected food and beverage items and their specifications into ordering information. By performing image processing and deep learning judgment through the interactive device, the system can recognize the user's hand gesture, thereby activating and operating the ordering process in a contactless manner, reducing physical contact and improving the convenience of interaction.

Claims

1. A smart gesture ordering system, comprising: a camera device capable of capturing a user image and transmitting the user image; and an interactive device coupled to the camera device, the interactive device having a database, a computing module, an audio module, and a display interface, the database containing an initiating hand gesture, an initiating command corresponding to the initiating hand gesture, a plurality of selectable hand gestures, a plurality of selectable commands corresponding to each of the selected hand gestures, a plurality of food items, and a plurality of item specifications corresponding to each of the food items; the computing module controlling the audio module to play an initiating voice, the computing module receiving the user image generated by the user according to the initiating voice, processing the user image to extract a hand image, the computing module using a deep learning model to determine whether the hand image matches the initiating hand gesture, and when the computing module determines that the hand image matches the initiating hand gesture, the computing module... The module obtains and retrieves each catering item from the database according to the start command, and displays each catering item on the display interface. The user inputs the user image through the camera device to select one of the catering items. The computing module receives and processes the user image, extracts the hand image from the user image, and obtains one of the matching hand gestures and one of the corresponding selection commands from the hand image through a deep learning model. The computing module obtains the specifications of each catering item according to the selection command, and displays the obtained specifications of each item on the display interface. The user can input the user image through the camera device to select the item specifications. The computing module obtains the specifications of the selected item through a deep learning model. The computing module integrates the selected catering item and the corresponding item specifications into order information.

2. The intelligent gesture ordering system as described in claim 1, wherein, The database also stores a confirmed hand gesture and a corresponding confirmation command. When the computing module uses a deep learning model to determine that the hand image matches the confirmed hand gesture and obtains the corresponding confirmation command, the computing module integrates the selected catering item and the corresponding item specification into the ordering information based on the confirmation command.

3. The intelligent gesture ordering system as described in claim 2, wherein, The hand image has multiple fulcrums and multiple levers. Each fulcrum corresponds to each finger joint and the junction of the hand and wrist. Each lever connects two adjacent fulcrums. The computing module uses a deep learning model to determine the positional relationship between each fulcrum, so as to obtain the matching starting hand posture, the selected hand posture, or the confirmed hand posture from the database.

4. The intelligent gesture ordering system as described in claim 3, wherein, The user image is a dynamic image. The computing module divides the user image into multiple stage user images according to the time sequence. The computing module performs image processing on each stage user image and generates the hand image according to the time sequence. The module also obtains the start hand gesture, the selection hand gesture, or the confirmation hand gesture from the database according to the time sequence. The computing module obtains the corresponding start command, the selection command, or the confirmation command from the obtained start hand gesture, the selection hand gesture, or the confirmation hand gesture.

5. The intelligent gesture ordering system as described in any one of claims 1 to 4, wherein, The camera device has a visual camera module and an auxiliary camera module. The visual camera module is used to capture the image of the user, and the auxiliary camera module is used to assist the visual camera module in capturing the image of the user in low light environment.

6. The intelligent gesture ordering system as described in claim 5, wherein, The visual photography module is a high-sensitivity stereo vision photography module, and the auxiliary photography module is an infrared sensing module.

7. The intelligent gesture ordering system as described in any one of claims 2 to 4 further comprises a cloud server coupled to the interactive device, wherein the interactive device is capable of transmitting the hand image obtained by image processing of the user image and the corresponding start hand gesture, selection hand gesture or confirmation hand gesture to the cloud server for recording, the cloud server is capable of classifying and storing the hand image according to the start hand gesture, selection hand gesture or confirmation hand gesture, and performing deep learning processing to generate optimized start hand gesture, selection hand gesture or confirmation hand gesture.

8. The intelligent gesture ordering system as described in claim 7, wherein, The cloud server will send the optimized start hand gesture, the selected hand gesture, or the confirmed hand gesture back to the database of the interactive device.

9. The intelligent gesture ordering system as described in claim 1, wherein, The interactive device is connected to a kitchen display system (KDS) via the Internet, and the computing module transmits the order information to the kitchen display system.

10. The intelligent gesture ordering system as described in claim 1 or 9, wherein, The computing module transmits the order information to the display interface for display.

Images