AI ordering assistant system capable of recognizing unstructured customer utterances through lightweight natural language processing

Abstract

The present invention relates to an AI order assistant system, and more specifically, to an AI order assistant system capable of receiving an order by receiving an unstructured voice order from a user, understanding the user's intent to generate an order form, and collecting information necessary for creating the order form by conversationally querying if there is missing order information, thereby automatically completing the final order form.

Description

Technology Field

[0001] The present invention relates to an AI order assistant system, and more specifically, to an AI order assistant system capable of receiving an order by receiving an unstructured voice order from a user, understanding the user's intent to generate an order form, and collecting information necessary for creating the order form by conversationally querying if there is missing order information, thereby automatically completing the final order form. Background Technology

[0003] Conventional unmanned ordering systems typically utilize a touch-based interface in the form of kiosks or table ordering, where users place orders by directly manipulating the screen.

[0004] While such unmanned ordering systems have been useful in terms of improving order accuracy and reducing labor costs, there are problems where the process of finding a menu or selecting an option becomes very complex when there is a wide variety of menus or options, leading to longer order times and even order failures for users unfamiliar with IT devices or first-time customers.

[0005] Furthermore, the UI / UX is not intuitive, causing inconvenience to digitally vulnerable groups, such as the elderly and people with disabilities, as well as general users.

[0006] Meanwhile, conventional speech recognition systems operate primarily on command-based simple responses and lack the processing capability for natural language inputs, including various speakers' pronunciations, colloquialisms, and dialects. Furthermore, due to the fixed-form Natural Language Processing (NLP) structure applied to entire sentences, they suffer from high computational loads and reduced real-time responsiveness.

[0007] Therefore, there is a need for a lightweight voice-based interface technology specialized for unmanned ordering environments that efficiently processes natural user voice input and enables the completion of an order by obtaining necessary information through conversational queries if information is missing. The problem to be solved

[0009] The present invention has been devised to solve the aforementioned problems, and the objective of the present invention is to provide an AI order assistant system capable of creating an order form and completing an order by receiving voice input from a user without a touch-type UI (User Interface).

[0010] In addition, another objective of the present invention is to provide an AI order assistant system capable of real-time voice recognition and two-way conversation by applying a lightweight natural language processing module that can operate in a cloud environment.

[0011] In addition, another objective of the present invention is to provide an AI order assistant system capable of creating order forms by recognizing the speech of various speakers, including colloquialisms, dialects, and differences in pronunciation speed, with high accuracy. means of solving the problem

[0013] To achieve the above objective, the present invention comprises a front-end device installed in a store where product ordering is required, and a back-end device connected to the front-end device via a communication network. The front-end device comprises: an order start trigger module that executes an order process when an order start voice is input from a customer; a voice recognition module that receives and recognizes voice related to an order from a customer; a front-end communication module that transmits voice data recognized by the voice recognition module to the back-end device; and a missing information query and order confirmation video output module that outputs a video requesting missing information from the customer or outputs an order confirmation video when essential information required for an order is missing. The back-end device comprises: a back-end communication module that receives voice data from the front-end device. The present invention provides an AI order natural language processing module characterized by including: a back-end communication module that receives voice data from the back-end communication module, converts the voice data into an order form, and transmits it to the front-end device through the back-end communication module, or transmits a query for missing information to the front-end device if information required for order form creation is missing.

[0014] In a preferred embodiment, the AI ​​order natural language processing module includes a sentence separation model that separates received voice data into morpheme units and determines whether each separated morpheme is continuous or separate within a sentence, thereby separating the voice data by sentence.

[0015] In a preferred embodiment, the AI ​​order natural language processing module includes an intent classification model that calculates the probability that an entity to be stored in data slots required for order form generation exists within each separated sentence, and predicts and classifies which data slot contains a speech intent.

[0016] In a preferred embodiment, the AI ​​order natural language processing module includes a named entity recognition model that identifies which of the data slots each morpheme within each separated sentence corresponds to as a named entity, and tags the morpheme having the found named entity as an entity with the corresponding tag among the data slots.

[0017] In a preferred embodiment, the AI ​​order natural language processing module includes an order generation model that, among the information tagged with entities and tags, distinguishes tags as labels and entities as values ​​to convert them into a data structure for generating an order form, and normalizes the converted entities with predetermined keywords to generate an order form.

[0018] In a preferred embodiment, the order form generation model checks for untagged empty slots among the data slots, generates an order form to verify the empty slots, and the checking of the empty slots is performed for each menu.

[0019] In a preferred embodiment, the AI ​​order natural language processing module includes a conversational transition model that generates query data to receive voice input for filling the empty slot and transmits the order form generated together with the avatar video or image to the missing information query and order confirmation video output module. Effects of the invention

[0021] The present invention has the following excellent effects.

[0022] According to the AI ​​order assistant system of the present invention, order forms can be created and orders completed by receiving voice input from the user without a touch-type UI (User Interface), which has the advantage of enabling even digitally vulnerable groups to easily perform non-face-to-face orders.

[0023] In addition, according to the AI ​​order assistant system of the present invention, sentence separation, intent classification, named entity recognition, and order form generation from voice data are performed using a lightweight natural language processing method, so order forms can be created in real-time through voice recognition and two-way conversation, and there is an advantage of being able to recognize order voice data with high accuracy even for voice utterances from various speakers, including colloquialisms, dialects, and differences in pronunciation speed.

[0024] In addition, according to the AI ​​order assistant system of the present invention, voice recognition accuracy can be improved by predicting the user's intent and filling slot data, and this intent prediction can be utilized to predict the intent of the next order, thereby having the advantage of continuously improving the accuracy of intent prediction. Brief explanation of the drawing

[0026] FIG. 1 is a drawing showing the usage environment of an AI order assistant system according to an embodiment of the present invention, FIG. 2 is a diagram illustrating the configuration of an AI order assistant system according to an embodiment of the present invention, FIG. 3 is a drawing showing an order start video of an AI order assistant system according to an embodiment of the present invention, FIG. 4 is a diagram showing a missing information query video of an AI order assistant system according to an embodiment of the present invention, FIG. 5 is a drawing showing an order confirmation video of an AI order assistant system according to an embodiment of the present invention, FIG. 6 is a drawing showing an order completion video of an AI order assistant system according to one embodiment of the present invention. Specific details for implementing the invention

[0027] The terms used in this invention have been selected to be as widely used as possible; however, in specific cases, terms have been arbitrarily selected by the applicant. In such cases, the meaning should be understood by considering the meaning described or used in the detailed description of the invention, rather than merely the name of the term.

[0028] Hereinafter, the technical configuration of the present invention will be described in detail with reference to preferred embodiments illustrated in the attached drawings.

[0029] However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Throughout the specification, the same reference numerals indicate the same components.

[0031] FIG. 1 is a diagram showing the usage environment of an AI order assistant system according to one embodiment of the present invention.

[0032] Referring to FIG. 1, an AI order assistant system (100) according to one embodiment of the present invention comprises a front-end device (110) installed in a store requiring product ordering in the form of a kiosk to receive voice orders from a user, and a back-end device (120) connected to the front-end device (110) via a communication network, installed at a remote location and managed by an administrator, which receives voice orders from the front-end device (110), creates an order form, and transmits the created order form to the front-end device (110) to complete the order.

[0033] That is, the back-end device (120) is a type of cloud system, so voice processing is possible quickly even if the front-end device (110) is a low-spec device.

[0034] However, the back-end device (120) and the front-end device (110) can be integrated into a single device in the form of an edge system.

[0035] In addition, the front-end device (110) does not have a separate touch display because it only needs to receive voice orders.

[0036] However, to improve order accuracy, a touch display is provided, allowing for the acceptance of touch orders in addition to voice orders.

[0037] FIG. 2 is a diagram illustrating the configuration of an AI order assistant system according to an embodiment of the present invention.

[0038] Referring to FIG. 2, the front-end device (110) is installed in a store and receives voice orders from a user.

[0039] Additionally, the front-end device (110) is shown in the form of a kiosk but can be provided in the form of a tablet PC and includes a display for outputting video, a microphone for receiving voice, and a speaker for outputting voice.

[0040] Additionally, the front-end device (110) includes an order start trigger module (111), a voice recognition module (112), a front-end communication module (113), and a missing information query and order confirmation video output module (114).

[0041] The above order start trigger module (111) activates the order process when the orderer's order start voice is input and outputs the order start screen shown in FIG. 3.

[0042] FIG. 3 shows the order start screen (200), and the order start screen (200) includes a guidance message (210) asking to say an order start voice (e.g., "I'll order"), a guidance video (220) that displays a video or image of the person receiving the order and guides the order as if it were being placed with a person, and a menu (230).

[0043] That is, when the user checks the order start screen (200) and says "I'll order," the order process is activated.

[0044] This is intended to reduce power consumption by preventing activation while in the order waiting state.

[0045] The voice recognition module (112) receives and recognizes voice related to an order from the orderer.

[0046] Additionally, the voice recognition module (112) can convert voice into text and, after recognizing it, transmit the text information to the back end device (120), and conversely, receive text information from the back end device (120), convert it into a voice signal, and output it.

[0047] The front-end communication module (113) transmits recognized voice data to the back-end device (120) or receives data from the back-end device (120).

[0048] In addition, the front-end communication module (113) can transmit and receive data through a telephone communication network or an internet communication network.

[0049] The above-mentioned missing information query and order confirmation video output module (114) outputs a video requesting the missing information by querying the customer when essential information required for the order is missing from the customer's order voice, or finally outputs an order confirmation video for order confirmation and payment.

[0050] That is, the above-mentioned missing information query and order confirmation video output module (114) performs a WebView function.

[0051] The back-end device (120) is connected to the front-end device (110) via a communication network and includes a back-end communication module (121) and an AI order natural language processing module (122).

[0052] The back-end communication module (121) is a communication module that receives voice data from the front-end device (110).

[0053] The AI ​​order natural language processing module (122) receives voice data from the back-end communication module (121), extracts objects necessary for an order from the voice data to create an order form, and transmits the order form created through the back-end communication module (121) to the front-end device (110).

[0054] Additionally, when essential information required for creating an order form is missing, the AI ​​order natural language processing module (122) transmits a query for the missing information from the orderer to the front-end device (110).

[0055] That is, the above AI order natural language processing module (12) enables the user to complete an order form and receive an order by conversing with the user in a conversational manner without a touch-type UI (User Interface).

[0056] Additionally, the AI ​​natural language processing module (122) comprises a sentence splitting model (122a), an intent classification model (122b), a named entity recognition model (122c), an order form generation model (122d), and a conversation transition model (122e).

[0057] In addition, the above models (122a, 122b, 122c, 122d, 122e) are each software having a specific function and enable the AI ​​order natural language processing module (122), which is a computer device, to perform the order form creation process.

[0058] The above sentence separation model (122a) separates the received voice data into morpheme units and determines whether each morpheme is continuous or separate within the sentence, and separates the sentence based on the morpheme corresponding to the separation, thereby separating the voice data by sentence.

[0059] If the above voice data is "Give me one large iced cappuccino and one cup of espresso," the sentence separation model (122a) divides the morphemes of the sentence into tokens as shown in the code of Table 1 below, predicts whether each token has the meaning of 'continuity' or 'separation', and separates the sentence based on the tokens predicted to be separated. In addition, in the above voice data, 'ice', 'cappuccino', 'large', 'geo', 'espresso', 'han', and 'jan' are each predicted as continuous tokens, and 'hana-rang' and 'jwo' are predicted as separation tokens, so that the sentence is finally separated into one sentence "Give me one large iced cappuccino" and one sentence "Give me one cup of espresso".

[0060] tokens = ['Ice', 'Cappuccino', 'Big', 'Ge', 'With one', 'Espresso', 'One', 'Cup', 'Give']# Split the sentence into tokens ss_split_preds = [0, 0, 0, 0, 1, 0, 0, 0, 1]# Predict whether each token is continuous (0) or split (1) sentence = ['Ice cappuccino big one', 'Give me an espresso']# Final sentence splitting

[0061] The above intention classification model (122b) calculates the probability that an entity to be stored in the data slots required for order form generation exists within each separated sentence, and classifies which data slot contains the utterance intention.

[0062] That is, the above intention classification model (122b) calculates the probability that each of the tokens within each separated sentence is an object to be stored in predetermined data slots for ordering, and performs the role of preferentially finding the object corresponding to the data slot with a high probability and storing it in the data slot.

[0063] In other words, the above intention classification model (122b) performs the role of finding the data slot corresponding to the most intentionally uttered token in the utterance sentence uttered by the user.

[0064] In addition, the above data slot may include 'menu name, quantity, payment, whether to eat indoors, size, temperature' as entity names required for an order, and the entity names may have a tag system tagged as 'B-MENU, B-QTY, B-PAY, B-INDOOR, B-OPTION_SIZE, B-OPTION_TEMP'.

[0065] Table 2 below shows an example of how the above-mentioned intention classification model (122b) classifies intentions. In the sentence spoken by the user, the probability that the user is the owner containing the size slot is predicted to be about 96%, and the user's speech intention can be predicted by ensuring that the entity name related to the size is found and saved when filling the data slot.

[0066] predict_intent = ('ORDER_SIZE', 0.9615566730499268)# Among the data slots resulting from predicting the intent of the user utterance, the probability that the order (ORDER_SIZE) includes the size slot is 96 percent.

[0067] The named entity recognition model (122c) finds which of the data slots corresponds to the named entity of each morpheme within each separated sentence, and tags the morpheme having the found named entity as an entity with the corresponding tag among the data slots.

[0068] Table 2 below is an example of the named entity recognition model (122c) recognizing and tagging named entities. Since the user's intention is 'size', the named entity 'large' to be stored in the size slot is found first, and other named entities are also tagged according to the tag system.

[0069] [NER with intent=ORDER_SIZE] -> [('Ice', 'B-OPTION_TEMP'), ('Cappuccino', 'B-MENU'), ('Large', 'B-OPTION_SIZE'), ('Buy', 'O'), ('One', 'B-QTY')]#Find Named Entities to populate data slots#Here, 'NER' stands for 'Named Entity Recognition', i.e., Named Entities

[0070] The above order form generation model (122d) separates the tags among the information tagged with objects and tags into labels and separates the objects into values, converts them into a data structure for generating an order form, and generates an order form by normalizing the converted objects with predetermined keywords.

[0071] In addition, the present invention utilizes a JSON structure as the data structure for generating the order form.

[0072] Table 4 below is an example of the conversion of the data structure for creating the order form above, where 'OPTION_TEMP', 'MENU', 'OPTION_SIZE', and 'QTY' are converted into labels, and 'Ice', 'Cappuccino', 'Big', and 'One' are converted into values, resulting in a data structure suitable for creating an order form.

[0073] entities = [{'label': 'OPTION_TEMP', 'value': 'Ice'}, {'label': 'MENU', 'value': 'Cappuccino'}, {'label': 'OPTION_SIZE', 'value': 'Large'}, {'label': 'QTY', 'value': 'One'}]#Separate by label and value and convert to JSON structure

[0074] In addition, the above order form generation model (122d) generates an order form by normalizing the converted object 'value' as shown in Table 5 below into a predetermined keyword for creating an order form.

[0075] For example, the 'Ice' object is normalized to 'ice' because it must be displayed as 'ice' on the order form.

[0076] cleaned = [{'label': 'OPTION_TEMP', 'value': 'ice'}, {'label': 'MENU', 'value': 'Cappuccino'}, {'label': 'OPTION_SIZE', 'value': 'large'}, {'label': 'QTY', 'value': 1}]#Entity (Keyword) Normalization (Cleansing)

[0077] Up to this point, we have shown an example of creating an order form for the sentence "one large iced cappuccino and". Table 6 below shows an example of creating an order form for a different sentence, "give me a cup of espresso," by following the same process as Tables 1 through 5 above.

[0078] predict_intent = ('ORDER_QTY', 0.94451225411223546)[NER with intent=ORDER_QTY] -> [('Espresso', 'B-MENU'), ('Han', 'B-QTY'), ('Jan', 'I-QTY'), ('Gwo', 'O')] entities = [{'label': 'MENU', 'value': 'Espresso'}, {'label': 'QTY', 'value': 'Han Jan'}]cleaned = [{'label': 'MENU', 'value': 'Espresso'}, {'label': 'QTY', 'value': 1}]

[0079] In addition, the above order form generation model (122d) restructures the slot data of the entire sentence into a JSON structure as shown in Table 7 below, checks whether empty slot data exists as shown in Table 8, creates an order form for checking empty slots as shown in Table 9, and checks the empty slots as shown in Table 10. In addition, the check for empty slots is performed for each menu.

[0080] Finally, regarding the menu, it can be seen that empty slots were found only for espresso, and the empty slots represent the temperature and size of the espresso menu.

[0081] restructure_order_items = [[{'MENU': 'Cappuccino', 'QTY': 1, 'OPTIONS': {'TEMP': 'ice', 'SIZE': 'large'}}], [{'MENU': 'Espresso', 'QTY': 1, 'OPTIONS': {}}]]#Restructure with extracted menu and slot information (JSON structure)

[0082] slot_info(before missing check) = {'slot': 'PAY', 'menu': None, 'resp_index': 2}#Check for empty slots, empty slots exist for the payment slot and the second menu.

[0083] ind_missing_slots = {'orders': [{'MENU': 'Cappuccino', 'QTY': 1, 'OPTIONS': {'TEMP': 'ice', 'SIZE': 'large'}}, {'MENU': 'Espresso', 'QTY': 1, 'OPTIONS': {}}], 'PAY': None, 'INDOOR': None}#Order form creation - for checking empty slots

[0084] missing_slots = [{'slot': 'TEMP', 'menu': 'Espresso'}, {'slot': 'SIZE', 'menu': 'Espresso'}]#Check for empty slots for each menu, (Size slot and Temperature slot found that are empty only for Espresso)

[0085] The above conversation transition model (122e) generates query data to receive voice input to fill the empty slot and transmits the order form generated along with the avatar video or image to the above missing information query and order confirmation video output module (114).

[0086] Table 11 below is an example of a query created for the temperature slot and size slot information of the 'Espresso' menu as an example of the above query data.

[0087] TEMP_2 : Would you like your espresso cold or hot?#AI query to fill empty slots

[0088] In addition, the above conversation transition model (122e) generates query data as shown in Table 12 below and transmits the order form generated along with the avatar video or image to the above missing information query and order confirmation video output module (114).

[0089] last_result : {'orderSheet': {'orders': [{'MENU': 'Cappuccino', 'QTY': 1, 'OPTIONS': {'TEMP': 'ice', 'SIZE': 'large'}, 'IMAGE_URL': 'http: / / 192.168.0.156:8000 / media / menu_image / menu_chino_ice.png', 'PRICE': 4500}, {'MENU': 'Espresso', 'QTY': 1, 'OPTIONS': {}, 'IMAGE_URL': 'http: / / 192.168.0.156:8000 / media / menu_image / menu_esso_hot.png', 'PRICE': 5500}], 'PAY': None, 'INDOOR': None}, 'responseMsg': 'How would you like your espresso temperature?', 'intent': 'TEMP', 'video_url': 'http: / / 192.168.0.156:8000 / media / video / TEMP_2.mp4'}#orderSheet : Order sheet based on current orders#IMAGE_URL : Order menu image and price information on the order sheet#responseMsg : Query message for missing information#intent : Intent of the response message - For predicting the next intent through algorithm learning to enhance prediction#vidio_url : Avatar video

[0090] Then, the above-mentioned missing information query and order confirmation video output module (114) outputs the missing information query video shown in FIG. 4.

[0091] Referring to FIG. 4, the missing information query video (300) includes a missing information query message (310), an order form (330) ordered so far, and an avatar video (320).

[0092] In addition, in the example of Table 12 above, information regarding the temperature and size of the espresso is missing, so the missing information query message is output as 'How would you like the espresso temperature?', and the size is output as a message to be entered sequentially after receiving voice input regarding the temperature.

[0093] In addition, it can be confirmed that temperature and size information is missing from the option display location (331) for the espresso menu ordered so far.

[0094] When all the missing information is received in this way, the conversation transition model finally outputs an order confirmation video that confirms the order through the missing information query and order confirmation video output module (114).

[0095] FIG. 5 shows an example of the order confirmation video (400), which includes an order confirmation message (410), an avatar video (420), a final order menu (430), a payment method (450), and a pickup method (whether to consume indoors) (460), and can be seen that the temperature and size information is filled in the option display location (440) of the missing espresso menu.

[0096] When the user watches the order confirmation video and responds by voice confirming the final order, the order completion video (500) shown in FIG. 6 is output.

[0097] Additionally, the order completion video (500) displays a payment request message (510) requesting payment according to the payment method, an avatar video (520), and a final order content (530) including an order number when payment is completed.

[0098] Therefore, according to the AI ​​order assistant system of the present invention, a user can place an order using only voice without touch input, and voice recognition accuracy can be improved by predicting intent and filling slot data. Furthermore, this intent prediction can be utilized to predict the intent of the next order, thereby continuously improving the accuracy of intent prediction.

[0100] As described above, the present invention has been illustrated and explained with reference to preferred embodiments, but it is not limited to the aforementioned embodiments, and various changes and modifications may be made by those skilled in the art within the scope of the invention without departing from the spirit of the invention. Explanation of the symbols

[0102] 100: AI Order Assistant System 110: Front-end Device 120: Back-end device 111: Order start trigger module 112: Voice recognition module 113: Front-end communication module 114: Missing Information Query and Order Confirmation Video Output Module 121: Backend Communication Module 122: AI Order Natural Language Processing Module 122a: Sentence splitting model 122b: Intent classification model 122c: Named Entity Recognition Model 122d: Order Form Generation Model 122e:Conversational Transfer Model

Claims

Claim 1 delete Claim 2 delete Claim 3 The system includes a front-end device installed in a store where product ordering is required, and a back-end device connected to the front-end device via a communication network. The front-end device includes: an order start trigger module that executes an order process when an order start voice is input from a customer; a voice recognition module that receives and recognizes voice related to an order from a customer; a front-end communication module that transmits voice data recognized by the voice recognition module to the back-end device; and a missing information query and order confirmation video output module that outputs a video requesting missing information from the customer or outputs an order confirmation video when essential information required for an order is missing. The back-end device includes: a back-end communication module that receives voice data from the front-end device. The AI ​​order natural language processing module includes: a back-end communication module that receives voice data and converts the voice data into an order form, transmits it to the front-end device through the back-end communication module, or transmits a query for missing information to the front-end device if information required for creating an order form is missing; wherein the AI ​​order natural language processing module includes a sentence separation model that separates the received voice data into morpheme units and determines whether each separated morpheme is continuous or separated within a sentence to distinguish the voice data by sentence, and an intent classification model that calculates the probability that an entity to be stored in data slots required for creating an order form exists for each separated sentence, predicts which data slot contains the utterance intent, and classifies the data. Claim 4 In claim 3, the AI ​​order natural language processing module comprises a named entity recognition model that identifies which of the data slots each morpheme within a separated sentence corresponds to as a named entity, and tags the morpheme having the found named entity as an entity with the corresponding tag among the data slots. Claim 5 In claim 4, the AI ​​order natural language processing module comprises an order form generation model that, among information tagged with entities and tags, distinguishes tags as labels and entities as values ​​to convert them into a data structure for generating an order form, and normalizes the converted entities with predetermined keywords to generate an order form. Claim 6 In claim 5, the order form generation model checks untagged empty slots among the data slots, generates an order form to verify the empty slots, and the checking of the empty slots is performed for each menu, characterized by an AI order assistant system. Claim 7 In claim 6, the AI ​​order natural language processing module comprises a conversational transition model that generates query data to receive voice input for filling the empty slots and transmits the order form generated together with the avatar video or image to the missing information query and order confirmation video output module.

Images