AI voice interactive ordering terminal system

Abstract

The application discloses an AI voice interactive meal ordering terminal system and relates to the technical field of voice dialogue, and comprises the following steps: an audio interface module collects meal ordering voice to perform voice activity segmentation; a voice understanding module performs language detection and standardization processing; a dialogue reasoning module generates a natural language response and a structured action list; an operation verification module verifies the legality of the structured action according to a menu knowledge base, and sends the action failing in the verification to a rollback action recovery module for repair; an order state management module applies the structured action passing the verification to a shopping cart state machine, executes constraint conditions, calculates a price, and generates an order summary; and a voice output module generates voice feedback in a target language, and through the combination of the structured action list and the shopping cart state machine, the conversion from voice to a deterministic order is realized, the consistency of an order state in a multi-round dialogue is ensured, and the fault tolerance is improved through rollback recovery when the action verification fails.

Description

Technical Field

[0001] This invention relates to the field of voice dialogue technology, specifically to an AI-powered voice-interactive food ordering terminal system. Background Technology

[0002] Currently, fast food restaurants, canteens, and other catering establishments primarily use human cashiers or touch-screen self-service terminals to process orders. Human cashiers rely on manpower, which can easily lead to queues and congestion during peak hours, and also incurs high long-term operating costs. Touch-screen self-service terminals can alleviate the burden on manpower and reduce queuing time to some extent, but they still have significant drawbacks in practical application. First, touchscreens, as public interactive devices, pose hygiene risks. Second, for elderly users, visually impaired users, or users unfamiliar with smart devices, the operation of the touch interface is difficult and inconvenient. Third, touch-screen ordering requires users to browse the menu step by step and add items one by one, which is inefficient and cannot meet users' needs for quick ordering with just a sentence.

[0003] Related technologies attempt to incorporate voice interaction capabilities to simplify the ordering process through natural language dialogue. Although voice ordering helps improve interaction efficiency, its widespread application in real restaurant environments still faces many technical obstacles. Real restaurant environments are subject to interference from multiple sources of noise, such as background music, kitchen equipment noise, and conversations among people. In addition, differences in language and accents among different users lead to a significant decrease in the performance of voice recognition and semantic understanding, making it difficult to reach a practical level. Furthermore, related technologies lack the ability to verify menu constraints in the catering industry, making it impossible to verify the legality of user instructions before execution or to proactively request clarification when constraints are not met. This restricts the large-scale application of voice ordering technology in real commercial scenarios.

[0004] Therefore, there is an urgent need for a voice ordering system that is noise-robust, adaptable to multilingual and multi-accent scenarios, and has intelligent rollback and recovery capabilities. Summary of the Invention

[0005] The purpose of this invention is to provide an AI voice-interactive food ordering terminal system, which aims to solve at least one problem existing in the prior art.

[0006] The technical solution of this invention is: an AI voice-interactive food ordering terminal system, comprising: The audio interface module is configured to acquire the initial order voice and perform voice activity segmentation on the initial order voice to obtain the target order voice; The speech understanding module is configured to perform language detection on the target order speech to obtain the target language, and to perform standardization processing on the target order speech to obtain standardized text; wherein, the standardization processing includes at least one of noise reduction processing, speech-to-text conversion and text normalization processing; The dialogue reasoning module is configured to generate natural language responses and a list of executable structured actions based on the standardized text, a preset menu knowledge base, and the current shopping cart status. The operation verification module is configured to perform legality verification on each structured action in the structured action list based on the menu knowledge base, and send the structured actions that pass the verification to the order status management module, and send the structured actions that fail the verification to the rollback action recovery module. The rollback action recovery module is configured to repair the structured action based on the standardized text, session context, shopping cart snapshot, and menu knowledge base when the structured action fails to be verified by the operation verification module. The order status management module is configured to apply the validated structured actions to the shopping cart state machine, execute constraints, calculate prices, and generate order summaries. The voice output module is configured to generate voice feedback information from the natural language response in the target language and output the voice feedback information.

[0007] In some embodiments of the present invention, the speech understanding module includes a speech enhancement unit and a menu terminology normalization unit; The speech enhancement unit is used to improve the speech signal-to-noise ratio in scenarios with background music, device operation sounds, and overlapping speech through noise suppression and target speech separation algorithms. The menu terminology standardization unit is used to map size descriptions, quantity descriptions, and attribute descriptions in user spoken expressions to standard menu codes, so that the dialogue reasoning module can generate the structured action list based on unified codes in multi-turn dialogues; The speech understanding module also includes a speech rate adaptive unit, which is configured to detect the speech rate of the user's speech and adjust the speech enhancement parameters in the speech enhancement unit and the silence detection threshold for speech activity segmentation in the audio interface module according to the detected speech rate, so as to adapt to users with different speech rates.

[0008] In some embodiments of the present invention, the structured action list adopts JSON, key-value pair, or object array format, and each structured action includes at least one of the following fields: action type field, product identifier field, order line identifier field, quantity field, attribute field, confidence field, evidence field, timestamp field, and priority field; wherein, the action type field includes at least adding product, deleting product, updating product, setting attribute, setting quantity, applying discount, canceling order, confirming shopping cart, requesting clarification, and triggering payment; the confidence field is used to characterize the degree of credibility of the structured action inferred; the evidence field is used to record the statement fragment, reasoning basis, or rule source that triggers the structured action; the priority field is used to determine the execution order when multiple structured actions exist simultaneously.

[0009] In some embodiments of the present invention, a binding relationship table is established between the structured action list and the shopping cart state machine, and the binding relationship table defines the mapping of different action types to state update operations in the shopping cart state machine; Among them, the action of adding a product is mapped to the state transition of adding an order line, the action of deleting a product is mapped to the state transition of removing an order line, the action of setting an attribute is mapped to the state transition of updating an order line attribute, the action of confirming the shopping cart is mapped to the state transition of moving from the ordering stage to the confirmation stage, and the action of triggering payment is mapped to the state transition of moving from the confirmation stage to the payment stage. The binding relationship table further includes conditional mapping rules, which map the same action type to different state transition paths according to the current shopping cart state, wherein the shopping cart state includes at least one of the following: total order amount, product type, or user attribute.

[0010] In some embodiments of the present invention, the rollback action recovery module is further configured to recover the structured action based on the most recently added product, the most recently modified product, the most recently clarified product, or the current focus order line when the target product is not explicitly specified in the user statement; The rollback action recovery module is further configured to cancel the structured action of the previous round when the user statement contains a negative expression of the previous round's instruction, and then apply the structured action of the current round.

[0011] In some embodiments of the present invention, the system further includes a supervision layer, which includes an anomaly determination rule module and a hierarchical processing module; The anomaly determination rules are used to detect at least one of the following anomalies: number of failed action validations, number of failed field validations, conflict between action and menu constraints, mismatch between action and current stage, price recalculation anomaly, and rollback repair failure. The hierarchical processing module is configured to: trigger a clarification request when the structured action is determined to be a minor anomaly; trigger a rollback action to restore when the structured action is determined to be a moderate anomaly; and trigger action rejection, shopping cart state rollback, or session protection interruption when the structured action is determined to be a severe anomaly.

[0012] In some embodiments of the present invention, the menu knowledge base is a multilingual menu knowledge base, including product definitions, attribute constraints and business rules, and further includes multilingual mapping data. The multilingual mapping data is used to map product names and attribute values ​​expressed in different languages ​​or mixed languages ​​to a unified internal identifier. The same product is mapped to the same internal product identifier in different languages ​​or mixed languages, and the same attribute value is mapped to a unified attribute constraint value in different languages ​​or mixed languages.

[0013] In some embodiments of the present invention, the system supports dual-mode deployment, namely local mode and cloud mode. In the local mode, the audio interface module, the voice understanding module, the dialogue reasoning module, the operation verification module, the rollback action recovery module, the order status management module, and the voice output module run on the self-service terminal device; In the cloud mode, the self-service terminal device collects the initial ordering voice and transmits it to the server for processing. The initial ordering voice includes at least a session identifier. The local mode and the cloud mode use the same structured action format and the same shopping cart status format.

[0014] In some embodiments of the present invention, the system also supports switching between the local mode and the cloud mode; When a network interruption or communication quality below a preset threshold is detected, the system switches from cloud mode to local mode and synchronizes the shopping cart status, current stage, dialogue history and the most recent valid action corresponding to the session to the local execution environment. Once the network is restored, the shopping cart changes, action history, and session status added during the local mode will be synchronized back to the cloud environment.

[0015] In some embodiments of the present invention, a session state storage module is also included, which is configured to generate and save a shopping cart state snapshot and a dialogue context vector in each round of dialogue. The rollback action recovery module is further configured to, when the structured action verification fails, call the most recent valid shopping cart state snapshot and reconstruct the user intent in conjunction with the dialogue context vector to generate a corrected structured action.

[0016] This invention achieves a robust transition from free speech to deterministic orders by using a structured action list, operation verification, rollback repair, and applying structured actions to the shopping cart state, while ensuring consistency of multi-turn dialogue states and system stability under environmental noise. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the AI ​​voice interactive food ordering terminal system provided in an embodiment of the present invention; Figure 2 This is a flowchart of the AI ​​voice interactive food ordering method provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0019] Reference Figure 1 This illustration shows an AI voice-interactive food ordering terminal system provided by an embodiment of the present invention. The system specifically includes: The audio interface module is configured to acquire the initial order voice and perform voice activity segmentation on the initial order voice to obtain the target order voice.

[0020] The initial ordering voice refers to the natural language statements uttered by the user during the ordering process, containing order-related information such as product name, quantity, specifications, and modification instructions. Voice activity segmentation refers to the process of detecting segments that actually contain the user's voice from a continuous audio stream and separating them from silent segments and background noise segments.

[0021] In a specific implementation of this invention, the audio interface module is responsible for receiving the initial order voice from the user and accurately identifying valid voice segments from the raw audio containing ambient noise. This module first acquires the user's voice signal through a microphone, then performs voice activity segmentation technology to automatically determine which segments in the audio stream belong to the user speaking and which belong to silence or background noise, thereby dividing the continuous audio into independent voice segments as target order voices for subsequent modules to process. This process ensures that only the content actually spoken by the user is sent to the voice understanding module, avoiding the waste of computing resources and noise interference caused by transmitting the entire audio stream.

[0022] During peak hours at fast-food restaurants, background music, kitchen equipment noise, conversations, and voices from nearby counters overlap. When a user orders food via voice commands at a self-service terminal, the audio interface module captures this audio through a microphone. Simultaneously, other customers' voices and background music are present in the environment. In one embodiment of this invention, speech segmentation technology analyzes the energy characteristics and spectral properties of the audio signal to determine in real time whether each audio segment belongs to the user's voice or environmental noise. It accurately identifies the time interval of the user's order voice, extracts the audio segments within that interval, and discards other voices and background noise from before and after. Even in noisy environments, the module can accurately locate the start and end boundaries of the user's voice, sending the clean speech segments to the subsequent speech understanding module for processing. For example, when a user says "I'd like a hamburger" to the self-service terminal, the audio interface module activates speech activity segmentation technology, analyzing the energy value of the audio signal in real time. When the energy exceeds a preset threshold and the spectral characteristics match the speech pattern, it determines that the user has started speaking and records the starting point. When the user pauses briefly after saying "hamburger," it detects a drop in energy and determines that the speech has ended. Ultimately, the module precisely extracts a 2.3-second effective speech segment from the 30-second original audio containing noise, discards the background noise and other people's voices in the remaining time period, and obtains a clean "Give me a hamburger" speech segment as the target order speech.

[0023] In this invention, the audio interface module reduces the amount of data processed by subsequent modules by performing speech activity segmentation before speech recognition, and avoids silent and noisy segments being misidentified as invalid commands. Accurate speech boundary detection reduces the interference of environmental noise on the recognition results, enabling the system to accurately capture the user's true commands in a noisy restaurant environment, thereby improving the overall reliability and response efficiency of voice interaction.

[0024] The speech understanding module 102 is configured to perform language detection on the target order speech to obtain the target language, and to perform standardization processing on the target order speech to obtain standardized text; wherein the standardization processing includes at least one of noise reduction processing, speech-to-text conversion and text normalization processing.

[0025] Among them, language detection refers to the automatic identification of the language used in the ordering voice after speech activity segmentation; standardization processing refers to converting the original speech signal into a standardized text format that can be directly used by subsequent modules; noise reduction processing refers to the technology of filtering out environmental background noise from the speech signal to improve speech clarity; speech-to-text refers to converting the speech content in the speech signal into the corresponding text sequence; and text normalization refers to converting the original text sequence output by speech-to-text into text that conforms to the standard format of the ordering scenario, including operations such as number transcription, synonym unification, and omission completion.

[0026] In a specific implementation of this invention, the speech understanding module receives user speech segments output by the audio interface module. First, it performs language detection to determine the user's language, which will be used by the subsequent speech output module. Then, it performs standardization processing on the same speech segment, converting the speech signal into standardized text. Standardization processing includes at least one of the following operations: noise reduction, speech-to-text conversion, and text normalization. Noise reduction is performed before speech-to-text conversion to improve the quality of the speech signal. Speech-to-text conversion converts the denoised speech signal into an initial text sequence. Text normalization formats the initial text sequence, for example, converting numerical terms into Arabic numerals, standardizing colloquial product aliases into standard product names, and completing omitted subjects or objects based on context. Finally, it outputs standardized text and target language information for use by the dialogue reasoning module.

[0027] When a user says "I want a hamburger" to the self-service terminal, the audio interface module sends the speech segment to the speech understanding module. The module first performs language detection, analyzing the acoustic features and phoneme distribution of the speech, identifying that the user is speaking Chinese, and marking the target language as Chinese. Then, it enters the standardization process. The first step is noise reduction, filtering out background music and device noise from the speech segment to obtain a relatively clean user speech signal. The second step is speech-to-text conversion, converting the denoised speech into the initial text "I want a hamburger". The third step is text normalization, determining that "hamburger" is a standard product name and does not require conversion, while recognizing "one" as quantity information, and standardizing the entire sentence into the expression "I want 1 hamburger". Finally, the module outputs a result with the target language in Chinese and the standardized text "I want 1 hamburger", which is then used by the dialogue reasoning module to generate structured actions.

[0028] In some optional embodiments of the present invention, the speech understanding module 102 may further include a speech enhancement unit and a menu terminology standardization unit; The speech enhancement unit is used to improve the speech signal-to-noise ratio in scenarios with background music, device operation noise, and speech overlap, through noise suppression and target speech separation algorithms; The menu terminology standardization unit is used to map size descriptions, quantity descriptions, and attribute descriptions in user spoken expressions to standard menu codes, so that the dialogue reasoning module can generate a structured action list based on unified codes in multi-turn dialogues; The speech understanding module also includes a speech rate adaptive unit, which is configured to detect the speech rate of the user's speech and adjust the speech enhancement parameters in the speech enhancement unit and the silence detection threshold for speech activity segmentation in the audio interface module according to the detected speech rate, so as to adapt to users with different speech rates.

[0029] In this invention, the speech enhancement unit refers to a functional module that processes the acquired speech signal to suppress environmental noise and improve the clarity of the target speech. The standard menu encoding refers to a unified internal identifier assigned to each item, attribute, and specification in the menu. The silence detection threshold for speech activity segmentation refers to the energy or feature threshold used by the audio interface module to determine the start and end of a speech segment during speech activity segmentation; when the duration of silence exceeds this threshold, the sentence is considered to have ended. Speech enhancement parameters refer to the set of adjustable coefficients in the speech enhancement unit that control the noise suppression intensity and the target speech extraction effect.

[0030] In some optional embodiments of the present invention, the speech understanding module further includes three components: a speech enhancement unit, a menu terminology normalization unit, and a speech rate adaptive unit. The speech enhancement unit addresses three common interference scenarios in restaurant environments: background music, kitchen equipment noise, and overlapping speech from multiple people. It applies noise suppression and target speech separation algorithms to extract clean user speech signals from mixed audio, improving the speech signal-to-noise ratio. After speech-to-text conversion, the menu terminology normalization unit maps size, quantity, and attribute descriptions in the user's spoken expressions to standard menu codes, unifying different spoken expressions into the same internal identifier. This allows the dialogue reasoning module to generate structured action lists based on the unified code during multi-turn dialogues. The speech rate adaptive unit detects the user's speech rate in real time and performs two adjustments based on the detection results: adjusting the speech enhancement parameters in the speech enhancement unit to adapt to changes in speech features at different speech rates, and adjusting the silence detection threshold in the audio interface module for speech activity segmentation to adapt to differences in pause lengths between sentences at different speech rates.

[0031] For example, during peak hours, background music plays continuously in the restaurant, kitchen equipment emits periodic noise, and conversations between customers at nearby counters overlap with the user's voice. The user speaks "Hello, I want to order" at the self-service terminal, speaking too quickly. The speech rate adaptive unit first detects that the user's speech rate is 220 syllables / min, exceeding the normal range. This unit sends a command to the audio interface module to lower the silence detection threshold for speech activity segmentation from the default 300ms to 120ms, preventing extremely short pauses between sentences by fast-speaking users from being misinterpreted as sentence endings and causing incorrect speech truncation. Simultaneously, the speech rate adaptive unit adjusts the speech enhancement parameters in the speech enhancement unit, reducing the frame length from 25ms to 15ms and the gain smoothing coefficient from 0.8 to 0.5, enabling the speech enhancement unit to quickly track drastic changes in speech energy under fast-speaking conditions. When the user then says "A hamburger and a large Coke," the speech activity segmentation uses a new silence detection threshold to segment the acquired order speech. The speech enhancement unit in the speech understanding module receives the speech segment, uses adjusted speech enhancement parameters, and employs a target speech separation algorithm to extract the clean speech signal of "A hamburger and a large Coke" from the mixed audio. The speech-to-text function converts the enhanced speech into the text "A hamburger and a large Coke." The menu terminology normalization unit maps "hamburger" to product code BG001, "Coke" to product code KL002, and "large" to size code L, outputting a unified encoding format for the dialogue reasoning module to generate structured actions.

[0032] The dialogue reasoning module 103 is configured to generate natural language responses and a list of executable structured actions based on standardized text, a preset menu knowledge base, and the current shopping cart status.

[0033] The preset menu knowledge base refers to a pre-built structured dataset containing product definitions, attribute constraints, default values, business rules, and multilingual mapping data. The current shopping cart status refers to the set of confirmed and unconfirmed order lines in the current session, including a structured representation of each order line's product identifier, quantity, attributes, unit price, and total. Natural language responses refer to user-generated replies in natural language, used to confirm operation results or clarify inquiries. The structured action list refers to a sequence of machine-executable action instructions, each encoded in a predefined data format, which can be directly parsed and executed by downstream modules.

[0034] In a specific implementation of this invention, the dialogue reasoning module receives standardized text output by the speech understanding module, combines it with a preset menu knowledge base and the current shopping cart status, and simultaneously generates two different types of output. The first output is a natural language response, used for interactive communication with the user, such as confirming added items or inquiring about missing information. The second output is a structured action list, where each action is encoded according to a predefined machine-readable format and can be directly parsed by the operation verification module. During the same reasoning process, the dialogue reasoning module simultaneously generates user-oriented natural language and machine-oriented structured instructions. Both outputs are generated based on the same input information, ensuring that the content of the natural language response remains semantically consistent with the structured action list.

[0035] Continuing with the previous example, when the user says "Give me a hamburger, large," the speech understanding module outputs the standardized text "Give me 1 hamburger, large." The dialogue reasoning module obtains this standardized text, accesses the preset menu knowledge base to confirm that "hamburger" is a valid product and "large" is a valid size attribute for a hamburger, and simultaneously checks that the current shopping cart is empty. After performing reasoning, the module generates two outputs: a natural language response of "A large hamburger has been added for you," and a structured action list containing an action encoded as "Add Product," "Hamburger," "Quantity," and "Large." It should be noted that in this invention, the natural language response and the structured action list are semantically completely corresponding; the confirmation information seen by the user and the operation instructions executed by the machine originate from the same reasoning result.

[0036] In this embodiment, the dialogue reasoning module simultaneously generates natural language responses and a structured action list. The user-oriented natural language responses ensure the naturalness and comprehensibility of the interaction, while the machine-oriented structured action list ensures the determinism and verifiability of order updates. Both outputs are generated based on the same input and reasoning logic, avoiding inconsistencies between the natural language responses and the actual actions performed.

[0037] In some embodiments of the present invention, the structured action list adopts JSON, key-value pair, or object array format, and each structured action includes at least one of the following fields: action type field, product identifier field, order line identifier field, quantity field, attribute field, confidence field, evidence field, timestamp field, and priority field; wherein, the action type field includes at least adding product, deleting product, updating product, setting attribute, setting quantity, applying discount, canceling order, confirming shopping cart, requesting clarification, and triggering payment; the confidence field is used to characterize the degree of credibility of the structured action inferred; the evidence field is used to record the statement fragment, reasoning basis, or rule source that triggers the structured action; the priority field is used to determine the execution order when multiple structured actions exist simultaneously.

[0038] JSON refers to a lightweight data exchange format that uses key-value pairs and a structured data representation. Key-value pair format refers to data organization where keys and corresponding values ​​appear in pairs. Object array format refers to a data collection composed of multiple structured objects arranged in a linear order. Confidence field is a field used to record the degree of credibility of a structured action when it is inferred; its value is a floating-point number between 0 and 1. Evidence field is a field used to record the original statement fragment, reasoning basis, or rule source that triggered the structured action. Priority field is a field used to determine the execution order when multiple structured actions exist simultaneously.

[0039] In a specific implementation of this invention, the structured action list is encoded using any of the following formats: JSON, key-value pairs, or object arrays. Each structured action includes at least one or more of the following fields: action type, product identifier, order line identifier, quantity, attribute, confidence level, evidence, timestamp, and priority. The action type field defines the operation category of the action, including at least ten types: adding a product, deleting a product, updating a product, setting attributes, setting quantity, applying discounts, canceling an order, confirming the shopping cart, requesting clarification, and triggering payment. The confidence level field records the credibility of the inference result of the dialogue reasoning module for this action; a higher value indicates a more reliable inference. The confidence level is specifically derived from a weighted comprehensive calculation of speech recognition confidence, semantic matching confidence, context consistency confidence, and rule matching confidence. The evidence field records the original fragment of the user's statement that triggered the action, the specific basis for the reasoning module's judgment, or the source of the rules invoked.

[0040] In a specific implementation of this invention, the generated structured action list undergoes field validation. Field validation includes at least: checking whether required fields are missing, whether field types match, whether field values ​​fall within the allowed range, whether there are logical conflicts between fields, and whether fields are consistent with product definitions in the menu knowledge base. For example, when the action type is setting an attribute, the attribute field should exist; when the action type is setting a quantity, the quantity field should exist and be a valid positive integer; when the action type is adding a product, the product identifier field should match the product entry in the menu knowledge base. In a specific implementation of this invention, action priority allocation is also included. In one embodiment, cancellation actions are executed before modification actions, confirmation actions are executed after product addition and attribute modification actions, and payment trigger actions are executed after the shopping cart is complete and constraints are satisfied. When multiple actions act on the same order line simultaneously, they are prioritized and executed according to the action priority field, timestamp field, and context stage.

[0041] In one embodiment of the present invention, structured actions can be JSON encoded in the following format.

[0042] Operation: ={ "Action Type": String, / / Values ​​include: "Add Item", "Remove Item", "Update Item", "Set Attributes", "Set Quantity", "Apply for Discount", "Cancel Order", "Confirm Cart", "Request Clarification", "Trigger Payment" "item_id": String, / / Product identifier field "line_id": String, / / optional, order line identifier field "quantity": Integer, / / optional, quantity field "attributes": Dictionary, / / optional, attribute fields "confidence": Floating-point number, / / optional, confidence level field, value range 0-1 "evidence": {"discourse span": "...", "reason": "..."}, / / optional, evidence field "timestamp": Time, / / timestamp field "priority": Integer, / / Priority field } In one embodiment of the present invention, the shopping cart status can be encoded in JSON format as follows.

[0043] CartState:={ "session_id": a string. "Language": string, "Stage": String, / / Values: "Order", "Clarify", "Confirm", "Payment", "Final Confirmation" "Number of lines": [{ "line_id": a string. "item_id": a string. "name": a string, "quantity": an integer. "attributes": dictionary entry. "unit_price": A numerical value. "line_total": numerical value}], "Total": {"Subtotal": value, "Tax": value, "Discount": value, "Total": value}, "Requirements to be processed": Array "History": {"Recent Actions": array, "Recent Posts": array}} When a user says "Give me a large Coke, no ice" to the self-service terminal, the dialogue reasoning module analyzes the statement and generates a structured action. The action type field is "Add Product", the product identifier field is "KL002", the quantity field is 1, the attribute fields are "Large" and "No Ice", the confidence field is 0.94, the evidence field is the statement fragment "Give me a large Coke, no ice" and the reason "speech recognition confidence 0.96, semantic matching confidence 0.95 (because large and no ice are both valid attributes of Coke), context consistency confidence 0.91, weighted sum is 0.94", the timestamp field is 2025-05-16T10:30:15.123Z, and the priority field is 1. The structured action is encoded in JSON format as follows: {"Type": "Add Product", "item_id": "KL002", "quantity": 1, "attributes": {"Size": "Large", "Temperature": "No Ice"}, "confidence": 0.94, "evidence": {"Speech Span": "Give me a large Coke, no ice", "Reason": "Speech recognition confidence 0.96, semantic matching confidence 0.95, context consistency confidence 0.91, weighted sum 0.94"}, "timestamp": "2025-05-16T10:30:15.123Z", "priority": 1}. The corresponding shopping cart status is encoded in JSON format as follows: {"session_id": "sess_20250516_001", "Language": "zh-CN", "Stage": "Ordering"} "Number of rows": [{"line_id": "L001", "item_id": "KL002", "name": "Cola", "quantity": 1,"attributes": {"size": "Large", "temperature": "No ice"},"unit_price": 8.00,"line_total": 8.00}],"Total": {"Subtotal": 8.00,"Tax": 0.00,"Discount": 0.00,"Total": 8.00},"Requests to be processed": [],"History": {"Recent actions": ["Added product_KL002_Large_No ice"],"Recent comments": ["Give me a large Coke, no ice"]}}. In the above example, the confidence score is calculated as the arithmetic mean of the speech recognition confidence score, semantic matching confidence score, and context consistency confidence score. In other embodiments of the present invention, the specific calculation formula for the confidence score field can also be C total =w1×C asr +w2×C nlu +w3×Cctx , where C asr C represents the speech recognition confidence score, the raw confidence score returned by the speech-to-text engine. It reflects the accurate probability of converting the audio signal into text, and its value ranges from [0,1]. nlu The semantic matching confidence score reflects the degree of closeness between the standardized text and the products / attributes in the menu knowledge base. Specifically, when the semantics are an exact match, such as "large Coke" completely matching the menu, C... nlu =1.0, when the semantics are fuzzy matching, such as "large" mapping to "large cup", C nlu =0.8, C when not matched / revert required nlu =0.3, C ctx Context consistency confidence reflects the logical compatibility between the current action and the current stage of the shopping cart state machine. Specifically, when the logic is reasonable, such as adding items during the ordering stage, then C... ctx =1.0, the logic is questionable, for example, attempting to modify the quantity during the payment phase, C ctx =0.5; Logical conflict, such as when deleting a non-existent product, C ctx =0.1, and the weights satisfy w1+w2+w3=1. For example, when emphasizing speech quality and semantic accuracy, w1=0.4, w2=0.4, and w3=0.2 are set. Furthermore, the weight coefficients are not fixed values; specifically, they can be dynamically adjusted according to the current environmental noise level. When the environment is extremely noisy, such as when the environmental signal-to-noise ratio is <10dB, the speech layer weight is reduced to 0.2, and the semantic layer weight is increased to 0.6 to offset the impact of speech recognition errors. For example, if a user says "Give me a hamburger, a large one" in a noisy environment, due to background music, the speech engine returns a low confidence level, C. asr A value of 0.7 maps "hamburger" to code BG001 and "large" to size L, which is a fuzzy match. (C) nlu The value is 0.8; the current stage is "ordering food," so the logic for adding products is reasonable. (C) ctx The value is set to 1.0. Assuming the current weights are w1=0.4, w2=0.4, w3=0.2, C total =0.4×0.7+0.4×0.8+0.2×1.0=0.28+0.32+0.2=0.80.

[0044] The user continues to say "Give me a hamburger" into the terminal. The second structured action is encoded as follows: Action type field value is "Add product", product identifier field value is "BG001", quantity field value is 1, attribute field value is empty, confidence field value is 0.88, evidence field value is the sentence fragment "Give me a hamburger", timestamp field value is 2025-05-16T10:30:15.456Z, priority field value is 2. The corresponding structured action is represented as: {"Type": "Add product", "item_id": "BG001", "quantity": 1, "attributes": {}, "confidence": 0.88, "evidence": {"speech span": "Give me a hamburger", "reason": "speech recognition confidence 0.90, semantic matching confidence 0.92 (hamburger precisely matches menu product), context consistency confidence 0.82, weighted sum 0.8 8"}, "timestamp": "2025-05-16T10:30:15.456Z", "priority": 2}, the JSON encoding of the corresponding shopping cart status at this time is: {"session_id": "sess_20250516_001", "language": "zh-CN", "stage": "order", "order line list": [{"line_id": "L001", "item_id": "KL002", "na "me": "Cola", "quantity": 1, "attributes": {"size": "Large", "temperature": "No ice"}, "unit_price": 8.00, "line_total": 8.00}, {"line_id": "L002", "item_id": "BG001", "name": "Burger", "quantity": 1, "attributes": {"size": "Regular"}, "unit_price": 25.00, "line_total": 25.00}], "Total": {"Subtotal": 33.00, "Tax": 0.00, "Discount": 0.00, "Total": 33.00}, "Requests to be processed": [], "History": {"Recent actions": ["Added product_KL002_Large_No ice", "Added product_BG001"], "Recent comments": ["Give me a large Coke, no ice", "Give me a hamburger"]}}.

[0045] After receiving these two actions, the operation verification module determines whether to execute the first action first and then the second action based on the priority field. If an exception occurs during subsequent execution, the statement fragment and reasoning in the evidence field can be used by the rollback action recovery module to trace the original user input and reasoning process. The confidence field can be used by the supervision layer to determine whether to initiate clarification and confirmation to the user. When the confidence level is lower than the preset threshold of 0.6, the clarification process is triggered.

[0046] In some embodiments of the present invention, a binding relationship table is established between the structured action list and the shopping cart state machine. The binding relationship table defines the mapping of different action types to state update operations in the shopping cart state machine. Among them, the action of adding a product is mapped to the state transition of adding an order line, the action of deleting a product is mapped to the state transition of removing an order line, the action of setting an attribute is mapped to the state transition of updating an order line attribute, the action of confirming the shopping cart is mapped to the state transition of moving from the ordering stage to the confirmation stage, and the action of triggering payment is mapped to the state transition of moving from the confirmation stage to the payment stage. The binding relationship table further includes conditional mapping rules, which map the same action type to different state transition paths based on the current shopping cart state. The shopping cart state includes at least one of the following: total order amount, product type, or user attribute.

[0047] The binding relationship table refers to the data structure used to record the correspondence between action types and state update operations in the shopping cart state machine. State transition refers to the process by which the shopping cart state machine moves from one stage to another. Conditional mapping rules refer to the mapping rules that dynamically select different state transition paths based on the current shopping cart state.

[0048] In a specific implementation of this invention, a binding relationship table is established between the structured action list and the shopping cart state machine. This binding relationship table defines the mapping relationship between different action types and state update operations in the shopping cart state machine. Adding a product action maps to the "Add Order Line" state transition, i.e., creating a new order line in the shopping cart. Deleting a product action maps to the "Remove Order Line" state transition, i.e., deleting the specified order line in the shopping cart. Setting an attribute action maps to the "Update Order Line Attribute" state transition, i.e., modifying the attribute information of an existing order line. Confirming the shopping cart action maps to the "From Order Stage to Confirmation Stage," i.e., completing order construction and entering the verification process. Triggering payment action maps to the "From Confirmation Stage to Payment Stage," i.e., completing verification and entering the payment process. In a specific implementation of this invention, the binding relationship table further includes conditional mapping rules. Specifically, these rules can map the same action type to different state transition paths based on the total order amount, product type, or user attributes in the current shopping cart state. For example, the "Confirm Shopping Cart" action might map to the "Discount Recommendation Stage" instead of directly entering the confirmation stage when the total order amount exceeds a preset threshold.

[0049] For example, at a fast-food restaurant's self-service terminal, after a user places an order and says "Confirm Order," the dialogue reasoning module generates a structured action to confirm the shopping cart type. After the operation verification module verifies the action, it is sent to the order status management module. The order status management module queries the binding relationship table, first checking the total order amount in the current shopping cart status. If the total order amount is 80 yuan, below the 100 yuan threshold, the conditional mapping rule maps the confirm shopping cart action to the confirmation stage from the ordering stage. The system directly enters the order confirmation interface and announces, "Your order totals 80 yuan, please confirm." If the total order amount is 150 yuan, exceeding the 100 yuan threshold, this invention maps the action to the discount recommendation stage path using conditional mapping rules. The system first enters the discount recommendation interface and announces, "Your order has exceeded 150 yuan, would you like to use a discount coupon?" The user selects a discount before entering the confirmation stage. If the current shopping cart contains pre-ordered items, this invention maps the confirm shopping cart action to the delivery time confirmation stage based on the item type using conditional mapping rules. The system first asks the user for their desired delivery time before proceeding to the next step. If a user is a member and has more than 500 points, this invention uses conditional mapping rules to map the shopping cart confirmation action to the points redemption recommendation stage path based on the user's attributes. It first asks whether to use points redemption before entering the confirmation stage.

[0050] In this embodiment, the binding relationship table centrally manages the mapping relationship between action types and state transition paths, facilitating the addition of new action types or modification of existing mapping relationships. The conditional mapping rules in this embodiment enable the same action type to dynamically select different state transition paths based on the current shopping cart state. Conditional mapping based on total order amount guides users to enjoy discounts in high-value orders, improving user experience. Conditional mapping based on product type automatically inserts necessary confirmation steps for special types such as pre-ordered and limited-time products, avoiding order disputes caused by missing processes. Conditional mapping based on user attributes provides differentiated and personalized services to member users, enhancing the system's commercial value.

[0051] In some embodiments of the present invention, the menu knowledge base is a multilingual menu knowledge base, including product definitions, attribute constraints and business rules, and further includes multilingual mapping data. The multilingual mapping data is used to map product names and attribute values ​​expressed in different languages ​​or mixed languages ​​to a unified internal identifier. The same product is mapped to the same internal product identifier in different languages ​​or mixed languages, and the same attribute value is mapped to a unified attribute constraint value in different languages ​​or mixed languages.

[0052] The multilingual menu knowledge base refers to a structured knowledge set storing product definitions, attribute constraints, business rules, and multilingual mapping data. Product definitions refer to the basic information of each product in the menu, such as its identifier, name, description, and category. Attribute constraints refer to the allowed values ​​for attributes such as size, temperature, and toppings for each product, as well as the mutual exclusion or dependency relationships between attributes. Business rules refer to order-level constraints, including minimum order amount, delivery restrictions, and discount application conditions. Multilingual mapping data refers to a set of mapping relationships that associate product names and attribute values ​​expressed in different or mixed languages ​​with a unified internal identifier.

[0053] In a specific embodiment of the present invention, the menu knowledge base is constructed as a multilingual menu knowledge base. The knowledge base comprises three core components: product definitions, attribute constraints, and business rules, and further includes multilingual mapping data. The multilingual mapping data is configured to map product names and attribute values expressed in different languages or mixed languages to unified internal identifiers. Specifically, the present invention maps various names of the same product expressed in different languages, different accents or mixed languages to the same internal product identifier. For example, "汉堡" (hamburger) in Chinese, "burger" in English, "汉堡burger" in Chinese-English mixed language, "汉堡" in Cantonese and "汉堡" in Hokkien are all mapped to the same product code. Various descriptions of the same attribute value expressed in different languages or mixed languages are mapped to a unified attribute constraint value. For example, "大杯" (large size) in Chinese, "large" in English, and "large杯" in Chinese-English mixed language are all mapped to the size code for large size. Through the multilingual mapping data, different language expressions recognized by the speech understanding module are converted into unified internal identifiers. The dialogue reasoning module generates structured actions based on the unified internal identifiers, and the order status management module updates the shopping cart and calculates prices based on the unified internal identifiers.

[0054] For example, a self-service terminal of a fast food restaurant is deployed in an area where Chinese and English are mixed used, and multilingual mapping data is pre-constructed in the multilingual menu knowledge base. A first user utters "one hamburger" in Chinese, the speech understanding module recognizes the Chinese expression "汉堡", queries the multilingual mapping data and maps "汉堡" to the internal product identifier BG001. The dialogue reasoning module generates a structured action for adding a product type based on BG001. A second user utters "I want a burger" in English. The speech understanding module recognizes the English expression "burger", queries the multilingual mapping data and maps "burger" to the same internal product identifier BG001. The dialogue reasoning module also generates a structured action for adding a product type based on BG001. A third user utters "one burger" in mixed Chinese-English. The speech understanding module recognizes the mixed expression "burger", queries the multilingual mapping data and still maps "burger" to BG001, and the same structured action is generated. For attribute expressions, when a user utters three different expressions "大杯", "large" and "large杯", the multilingual mapping data maps all of them to the size code L. Three users order the same product in different languages or mixed languages, the finally generated internal product identifiers are completely consistent, and the constraint verification, price calculation and order generation logic executed by the order status management module are also completely consistent.

[0055] This invention employs a multilingual menu knowledge base, enabling automatic product and attribute recognition under different and mixed language inputs, eliminating the need to build a separate menu knowledge base for each language. Centralized management of multilingual mapping data ensures that adding new language support only requires expanding the mapping relationships, without modifying the core processing flow. Unified cross-language encoding rules guarantee completely consistent processing results for the same product across Chinese, English, and mixed language expressions in constraint validation, price calculation, and order generation, avoiding order logic differences caused by language variations.

[0056] The operation verification module 104 is configured to perform legality verification on each structured action in the structured action list based on the menu knowledge base, and send the structured actions that pass the verification to the order status management module, and send the structured actions that fail the verification to the rollback action recovery module.

[0057] The validity check refers to the process of checking each structured action against the constraint rules in the menu knowledge base. The checks include whether the product exists, whether the attributes are allowed, whether the quantity is valid, and whether required attributes are complete. A successful validity check means that the structured action has passed the validity check without any violations of constraint rules and can be executed safely. A failed validity check means that the structured action has at least one violation of constraint rules found during the validity check process, and the action cannot be executed directly.

[0058] In a specific implementation of this invention, the operation verification module receives a list of structured actions output by the dialogue reasoning module and independently performs a validity check on each structured action in the list. The verification process is based on predefined product definitions, attribute constraints, default values, and business rules in the menu knowledge base. For each structured action, the operation verification module determines whether it meets all constraints. If the structured action passes the validity check, it is sent to the order status management module for shopping cart update. If the structured action fails the validity check, it is sent to the rollback action recovery module for repair processing. The operation verification module does not modify the content of the structured actions; it only performs the judgment. Actions that pass the verification are directly sent to the execution path, while actions that fail the verification are directly sent to the repair path.

[0059] Continuing with the previous example, the dialogue reasoning module outputs a structured action: adding a product type, product identifier "hamburger," quantity "1," and attribute "large." The operation verification module performs a validity check based on the menu knowledge base. First, it checks if the product identifier "hamburger" exists in the menu knowledge base, which is found to be present. Second, it checks if the "large" attribute is a allowed value for "hamburger," which is also found to be allowed. Third, it checks if any required attributes for "hamburger" are missing, which is found to be none. Finally, it checks if the quantity "1" is a valid positive integer, which is also found to be valid. With all checks passed, the structured action is sent to the order status management module for execution. If another structured action is adding a product type, product identifier "hamburger," quantity "1," and attribute "medium spicy," but the attribute constraints for "hamburger" in the menu knowledge base do not include "medium spicy," the operation verification module determines that the verification fails and sends the action to the rollback action recovery module. If yet another structured action is setting an attribute type but omitting the product identifier field, the operation verification module detects a missing field, similarly determines that the verification fails, and sends the action to the rollback action recovery module.

[0060] The operation verification module performs legality checks before structured actions are executed, ensuring that only actions that meet the menu constraints can update the shopping cart. This fundamentally prevents order errors caused by issues such as non-existent products, disallowed attributes, or missing fields, thereby improving the accuracy of order data and the reliability of system execution.

[0061] The rollback action recovery module 105 is configured to repair structured actions based on standardized text, session context, shopping cart snapshot, and menu knowledge base when the structured action fails to be verified by the operation verification module.

[0062] Repair refers to supplementing, correcting, or rewriting structured actions that fail validation to ensure they meet legality requirements. Session context refers to the sequence of user statements, system responses, and interactions that occurred prior to the current dialogue round. A shopping cart snapshot is a static data set obtained by completely copying the shopping cart state at a specific moment, containing all order line information and total information at that time.

[0063] In a specific implementation of this invention, the rollback action recovery module receives the structured action that failed verification from the operation verification module and performs a repair operation on the action. The repair process is completed based on standardized text, session context, shopping cart snapshot, and menu knowledge base. Standardized text provides a standardized expression of the user's original input, session context provides a reference for historical interaction behavior, shopping cart snapshot provides a complete backup of the current order status, and menu knowledge base provides product attributes and constraint rules. Based on the above information, the rollback action recovery module infers the user's true intent, performs field completion, parameter correction, or action reconstruction on the structured action that failed verification, and outputs the repaired structured action. It should be noted that the repaired action needs to undergo legality verification again until it passes or is determined to be unrepairable.

[0064] Continuing with the previous example, when the user says "larger size," the dialogue reasoning module outputs a structured action: "Set attribute type" with the attribute value "large." However, the product identifier field is missing. The operation verification module detects the missing product identifier field and sends the action to the rollback action recovery module. The rollback action recovery module retrieves the standardized text "larger size," reads the conversation context to find that the user just added a medium-sized Coke in the previous round, reads the shopping cart snapshot to confirm that there is only one medium-sized Coke in the current cart and that the product allows size modification, and confirms that "large" is a valid size for Coke based on the menu knowledge base. Combining the above information, it infers that the user's intention is to change the medium-sized Coke to a large size, and generates a corrected structured action: "Set attribute type," with the order line identifier pointing to the order line containing the medium-sized Coke, and the attribute value being "large." The corrected action, after passing legality verification, is sent to the order status management module for execution.

[0065] The rollback action recovery module in this invention enables the ordering system to automatically infer the true intent and complete the action repair when the user's command is incomplete or ambiguous, avoiding interaction interruptions caused by direct refusal to execute due to missing commands. Based on a multi-source information fusion repair strategy using standardized text, session context, shopping cart snapshots, and menu knowledge base, the success rate in ambiguous command scenarios is significantly improved.

[0066] In some embodiments of the present invention, the rollback action recovery module is further configured to recover the structured action based on the most recently added product, the most recently modified product, the most recently clarified product, or the current focus order line when the target product is not explicitly specified in the user statement; The rollback action recovery module is also configured to undo the structured action of the previous round and then apply the structured action of the current round when the user statement contains a negative expression of the previous round's instruction.

[0067] Among them, "most recently added product" refers to the product corresponding to the most recent product addition operation in the current session at the current moment; "most recently modified product" refers to the product corresponding to the most recent attribute modification or quantity adjustment operation in the current session at the current moment; and "most recently clarified product" refers to the target product for which the system asked the user for confirmation in the previous clarification dialogue. "Currently focused order line" refers to the order line that the user is most likely to be operating on at present, inferred from the dialogue context and session state. "Negative expression" refers to words or phrases in the user's statement used to negate or correct the previous instruction, including "no," "not," "wrong," "said incorrectly," "inappropriate," "should be," etc.

[0068] In a specific implementation of this invention, when the target product is not explicitly specified in the user's statement, the rollback action recovery module recovers the complete structured action based on one or more of four candidate sources: the most recently added product, the most recently modified product, the most recently clarified product, or the currently focused order line. Specifically, the matching degree between each candidate source and the current user instruction can be evaluated sequentially according to priority, and the candidate with the highest matching degree can be selected as the target order line to complete the action repair. When the user's statement contains a negative expression of the previous instruction, the module first identifies that the user's intention is to correct the erroneous instruction of the previous round, performs an undo operation to roll back the changes to the shopping cart state caused by the previous round of structured actions, and then applies the structured action generated in the current round.

[0069] For example, a user first says "Give me a Coke," adding a medium-sized Coke to the shopping cart. The user then says "Change to a large," without explicitly specifying the target product. In this invention, the rollback action recovery module detects that the most recently added product is Coke, the most recently modified product is empty, the most recently clarified product is empty, and the currently focused order line points to the Coke order line. The rollback action recovery module restores "Change to a large" to a structured action with a set attribute type, the order line identifier points to the Coke order line, and the attribute value is "large." Suppose a user first says, "Give me a medium Coke," the dialogue reasoning module generates a structured action to add the product type. After the order status management module executes this action, a medium Coke is added to the shopping cart. The user then says, "No, I meant large," containing the negative expression "no." Upon detecting this negative expression, the rollback action recovery module first performs an undo operation, reversing the previous action of adding the medium Coke and removing it from the shopping cart. Then, the module applies the structured action of the current round and adds a large Coke to the shopping cart. Similarly, suppose a user first says, "Add a hamburger," adding a hamburger to the shopping cart. The user then says, "No, it's a chicken wrap," containing the negative expression "no." Upon detecting this negative expression, the rollback action recovery module first performs an undo operation, removing the hamburger added in the previous round. Then, based on the user's statement "chicken wrap," it generates a structured action to add the product type, identifying the product as a chicken wrap, thus completing the execution of the current round's instruction.

[0070] In a specific implementation of this invention, the rollback action recovery module may further include a library of typical invalid action repair steps, which defines the repair process for common error scenarios. For example, for scenarios such as simply saying "add" without specifying a product, vaguely modifying an order, missing quantity, conflicting attribute values, or the product not existing in the current menu, the sequential steps of candidate target search, knowledge base constraint comparison, focus product selection, default value completion, and request clarification can be defined respectively. Specifically, candidate target search refers to the process of searching for products or order lines that may match the user's intent in the shopping cart status or menu knowledge base; knowledge base constraint comparison refers to matching the attribute values ​​in the user's instruction with the allowed attribute values ​​defined in the menu knowledge base item by item; focus product selection refers to determining the most likely one from multiple candidate targets based on priority rules as the repair target; default value completion refers to reading preset default values ​​from the menu knowledge base and filling them in when the user's instruction lacks a required attribute; and request clarification refers to asking the user a follow-up question to obtain missing or vague information. For example, if a user says "change to a larger size" for a vague modification, this invention queries a typical invalid action repair step library. First, it performs a candidate target search, traversing all order lines in the shopping cart. Then, it performs a knowledge base constraint comparison, filtering out order lines that allow size modification. Next, it performs a focus product selection, prioritizing the most recently added product. Finally, it generates the repaired setting attribute action. As another example, if a user says "I'd like a hamburger" but doesn't specify a size, and the size is a required attribute in the menu knowledge base, this invention queries a typical invalid action repair step library. First, it performs default value completion, retrieving the default hamburger size as medium from the menu knowledge base. Then, it generates a repaired add product action, with the default size medium included in the attribute.

[0071] In some embodiments of the present invention, the system further includes a session state storage module, which is configured to generate and save a shopping cart state snapshot and a dialogue context vector in each round of dialogue. The rollback action recovery module is further configured to, when a structured action verification failure is detected, recall the most recent valid shopping cart state snapshot and reconstruct the user intent by combining the dialogue context vector to generate a corrected structured action.

[0072] The session state storage module persistently saves the shopping cart state and dialogue context vector during the dialogue. The shopping cart state snapshot refers to a static data set obtained by completely copying the shopping cart state at the end of a certain dialogue round, containing all order line information, total information, and current stage information at that moment. The dialogue context vector refers to the encoded historical interaction information before the current dialogue round as a fixed-dimensional numerical vector representation.

[0073] In a specific implementation of this invention, the system further includes a session state storage module. This module generates and saves a shopping cart state snapshot and a dialogue context vector in each round of dialogue. The shopping cart state snapshot records complete information about the shopping cart at the end of that round of dialogue, including all order lines, prices, and the current stage. The dialogue context vector encodes historical interaction information into a numerical vector for subsequent intent inference. The rollback action recovery module is further configured to, when a structured action verification failure is detected, recall the most recent valid shopping cart state snapshot, i.e., the shopping cart state after the most recent successful verification and execution. The rollback action recovery module combines the order information in this snapshot with the historical interaction information in the dialogue context vector to analyze the correlation between the user's current command and the historical state, infer the user's true intent, and generate a corrected structured action based on the inference result.

[0074] For example, in the first round of dialogue, the user says "Give me a hamburger." The session state storage module generates a shopping cart state snapshot A, containing a hamburger order line, and the dialogue context vector records the semantic features of this round of interaction. In the second round of dialogue, the user says "Give me a Coke." The session state storage module generates a shopping cart state snapshot B, containing two order lines: a hamburger and a Coke. The dialogue context vector is updated to include the semantic features of both rounds of interaction. In the third round of dialogue, the user says "Change to a large." The dialogue inference module generates a structured action that sets the attribute type, but the product identifier field is missing. The operation verification module fails to verify and sends the action to the rollback action recovery module. The rollback action recovery module calls the most recent valid shopping cart state snapshot B saved in the session state storage module to obtain information that the current shopping cart contains a hamburger and a Coke. The module analyzes the interaction records of the first two rounds in conjunction with the dialogue context vector, detecting that the user just added a Coke in the second round of dialogue and that the Coke has a size attribute, inferring that the user's intention is to change the Coke to a large size. The module generates a corrected structured action, with the order line identifier pointing to the Coke order line and the attribute value being "large." This action is executed after verification.

[0075] The order status management module 106 is configured to apply validated structured actions to the shopping cart state machine, execute constraints, calculate prices, and generate order summaries.

[0076] The shopping cart state machine refers to a finite number of states used to maintain the order state transition logic. In this invention, the shopping cart state machine specifically defines the transition paths between the ordering stage, confirmation stage, payment stage, and payment completion stage. Execution constraints refer to the structured actions that require the enforcement of mandatory attributes, allowed modifiers, availability rules, and order hierarchy rules defined in the menu knowledge base. Price calculation refers to the process of accumulating the unit price of each order line in the shopping cart and generating subtotals, taxes, discounts, and total amounts based on the base price and modifier price rules in the menu knowledge base. Order summary refers to the aggregated data containing details of all order lines in the current shopping cart, various price information, and the total order amount.

[0077] In a specific implementation of this invention, the order status management module receives the validated structured actions from the operation verification module and applies these actions to the shopping cart state machine to update the current shopping cart status. The operation verification module ensures that each action satisfies the business rules in the menu knowledge base, including automatic filling of required attributes, mutual exclusion attribute conflict detection, and availability verification. After the constraints are executed, the order status management module performs the corresponding state transition according to the type of the structured action. For example, adding a product triggers a state transition to add an order line, setting an attribute triggers a state transition to update order line attributes, and confirming the shopping cart triggers a state transition from the ordering stage to the confirmation stage. After each shopping cart status update, the module recalculates the price and generates subtotals, taxes, discounts, and totals based on the quantity and attributes of all order lines in the current shopping cart. Finally, the module generates an order summary for the voice output module to provide feedback to the user and for use by the subsequent payment module.

[0078] For example, a user has completed multiple rounds of ordering, and the current shopping cart contains one hamburger order line and one medium-sized cola order line. The user says "Change the cola to a large," and the dialogue reasoning module generates a structured action that sets the attribute type. The order line identifier points to the cola order line, and the attribute value is "large." This action is verified by the operation verification module and then sent to the order status management module. The module first executes the constraints, queries the menu knowledge base to confirm that "large" is a valid size attribute for cola, and confirms that there are no other required attributes missing for cola. Then, it applies the verified action to the shopping cart state machine, locates the cola order line corresponding to the order line identifier, and updates its attribute from "medium" to "large." This operation triggers the order line attribute update state transition. The module recalculates the price according to the pricing rules in the menu knowledge base: the hamburger price remains unchanged, the medium cola, originally priced at 5 yuan, is upgraded to a large cola for an additional 2 yuan, becoming 7 yuan. The shopping cart subtotal increases by 2 yuan, the tax is adjusted proportionally, and the total amount is updated synchronously. In addition, this module also generates an order summary, which includes the updated details of the two order lines and various price information.

[0079] In the present invention, the order status management module implements predictable update of order status by applying structured actions deterministically to the shopping cart state machine. The mandatory enforcement of constraints ensures that the final order satisfies all business rules in the menu knowledge base, and avoids invalid orders caused by missing or conflicting attributes. The mechanism of recalculating the price after each update ensures that the amount information is always consistent with the content of the shopping cart, and the generation of the order summary enables users to obtain clear and complete order confirmation information through the voice output module, which provides an accurate data basis for the payment process.

[0080] The voice output module 107 is configured to generate voice feedback information from natural language responses by using a target language, and output the voice feedback information.

[0081] Wherein, the target language refers to the language used by the user determined by the voice understanding module through language detection, and is used to control the language type of voice output. The voice feedback information refers to playable audio data obtained by converting natural language responses through speech synthesis technology.

[0082] In the specific implementation of the present invention, the voice output module receives the natural language response generated by the dialogue reasoning module, and simultaneously obtains the target language detected by the voice understanding module, calls a speech synthesis engine of the corresponding language based on the target language, and converts the text content of the natural language response into voice feedback information. The conversion process includes three stages: text analysis, prosody prediction and acoustic synthesis, which generates natural and smooth voice signals. After completing the speech synthesis, the voice feedback information is played out through the speaker of the terminal device or the audio output channel of the client device, so that the user can hear the voice reply from the system.

[0083] By way of example, the voice understanding module detects that the user uses Chinese, and marks the target language as Chinese. The dialogue reasoning module generates a natural language response "One hamburger has been added for you" based on the user input "I want a hamburger". The voice output module receives that the target language is Chinese and the natural language response is "One hamburger has been added for you", calls a Chinese speech synthesis engine, and converts this text into a voice signal with a duration of about 1.5 seconds. The module plays the voice signal through the speaker of the self-service terminal, and the user hears the voice feedback "One hamburger has been added for you". If the voice understanding module detects that the user uses English, the target language is marked as English, the natural language response generated by the dialogue reasoning module is "A hamburger has been added to your cart", and the voice output module calls an English speech synthesis engine to generate the corresponding English voice feedback.

[0084] The voice output module of this invention allows users to obtain order confirmation information through hearing without reading the screen, reducing the visual attention required during the ordering process. This is especially suitable for visually impaired users or those who have difficulty viewing screens. Speech synthesis based on the target language enables users to receive order feedback in their preferred language, improving the user experience.

[0085] In some embodiments of the present invention, the system supports dual-mode deployment, namely local mode and cloud mode. In local mode, the audio interface module, voice understanding module, dialogue reasoning module, operation verification module, rollback action recovery module, order status management module, and voice output module run on the self-service terminal device; In cloud mode, the self-service terminal device collects the initial ordering voice and transmits the initial ordering voice to the server for processing, wherein the initial ordering voice includes at least a session identifier; Both local and cloud modes use the same structured action format and the same shopping cart status format.

[0086] In this context, "local mode" refers to a working mode where all functional modules run locally on the self-service terminal device. "Cloud mode" refers to a working mode where the client device is only responsible for audio capture and playback, while the core processing module runs on a remote server. A self-service terminal device refers to a hardware device integrating a microphone, speaker, and touchscreen. A session identifier is a string used to uniquely identify a single ordering session, used to associate audio data and status information belonging to the same session between the client and server.

[0087] In a specific implementation of this invention, the system supports dual-mode deployment: local and cloud-based. In local mode, the aforementioned audio interface module, speech understanding module, dialogue reasoning module, operation verification module, rollback action recovery module, order status management module, and speech output module all run locally on the self-service terminal device. User speech completes the entire process of acquisition, understanding, reasoning, verification, status update, and speech feedback locally on the terminal, without requiring a network connection. In cloud mode, the client device is responsible for acquiring user audio and transmitting the audio data along with the session identifier to a remote server. The speech understanding module, dialogue reasoning module, operation verification module, rollback action recovery module, and order status management module running on the server perform processing. The server returns the processing results to the client device, where the speech output module generates speech feedback. Both local and cloud modes use the same structured action format and the same shopping cart status format. The field definitions for structured actions and the data structure for the shopping cart status are completely identical in both modes.

[0088] In one embodiment of the invention, the self-service terminal can be a hardware device deployed inside a restaurant, integrating a microphone, speaker, and touch screen. For example, a chain fast-food brand deploys self-service terminals in its stores. When the network within the store is stable, the self-service terminal operates in cloud mode, collecting user voice data and sending it to a cloud server for recognition and inference, utilizing a large-scale language model in the cloud to obtain more accurate semantic understanding. When the store network experiences fluctuations or interruptions, the self-service terminal automatically switches to local mode, with its built-in lightweight voice understanding and dialogue inference modules taking over processing, ensuring uninterrupted ordering for the user. In another embodiment of the invention, the self-service terminal can also include devices brought by the user or provided by the restaurant, equipped with audio collection and network communication capabilities, including smartphones and portable terminals. That is, the invention can simultaneously provide a mobile ordering application. A user can use the mobile ordering application at home, with the phone operating in cloud mode. After collecting user voice data, it is sent to a cloud server for processing via a mobile network. The server generates a structured action format, which is a JSON object composed of action type fields, product identifier fields, quantity fields, attribute fields, etc. The shopping cart status format is a JSON object containing an order line list, total amount, and current stage. The user then went to the restaurant and used the self-service terminal deployed in the restaurant to continue ordering food. The terminal ran in local mode, and the generated structured action format and shopping cart status format were completely consistent with the cloud mode. The user had a consistent interactive experience on both devices.

[0089] In some embodiments of the present invention, the system also supports switching between local mode and cloud mode; When a network interruption or communication quality below a preset threshold is detected, the system switches from cloud mode to local mode and synchronizes the shopping cart status, current stage, dialogue history and the most recent valid action corresponding to the session to the local execution environment. Once the network is restored, the shopping cart changes, action history, and session status during the local mode will be synchronized back to the cloud environment.

[0090] Among them, "communication quality below the preset threshold" refers to a state where indicators such as network latency, data packet loss rate, or bandwidth drop below a pre-set quality lower limit. Dialogue history refers to a complete time-series record of user statements, system responses, and interactive behaviors that have occurred in the current session. The most recent valid action refers to the last structured action in the current session that was verified by the operation verification module and successfully applied to the shopping cart state machine.

[0091] In a specific implementation of this invention, the system supports switching between local mode and cloud mode. When the system is running in cloud mode, it continuously monitors the network status between the client device and the cloud server. When a network interruption or communication quality below a preset threshold is detected, the system automatically triggers a switch from cloud mode to local mode. During the switch, the system first synchronizes the shopping cart status, current stage, dialogue history, and the most recent valid action corresponding to the current session from the cloud environment to the local execution environment, enabling the local mode to continue processing user requests from the moment of cloud interruption rather than starting from scratch. After the switch is completed, the system continues to run in local mode, with all functional modules executed on the self-service terminal device. Once the network is restored, the system synchronizes any new shopping cart changes, action history, and session status added during the local mode back to the cloud environment, ensuring that the state of the cloud environment is consistent with the latest state of the local execution environment, providing complete state data for any subsequent possible switch or cloud data analysis.

[0092] For example, a fast-food restaurant's self-service terminal operates in cloud mode, and a user begins ordering. The cloud server records the current session's shopping cart status, which already contains a hamburger and a medium-sized Coke. The current stage is the ordering stage, and the dialogue history includes two rounds of user commands, with the most recent valid action being the addition of a Coke. Just as the user says, "Change the Coke to a large," the store's network suddenly drops. The system detects a network connection timeout, determines a network interruption, and automatically triggers a switchover process. The system synchronizes the shopping cart status, the current stage (ordering stage), the dialogue history, and the most recent valid action to the local execution environment. After the switchover, local mode takes over. The user continues to say "large," and the rollback action recovery module in the local environment, combined with the synchronized shopping cart status, identifies the most recent valid action as adding a Coke, corrects "large" by setting the attribute action to the Coke order line, and completes the Coke size modification. Five minutes later, the network is restored, and the system synchronizes the shopping cart changes made during local mode, the action history of changing the medium Coke to a large, and the updated session status back to the cloud environment. The state of the cloud environment remains consistent with the local execution environment.

[0093] This embodiment switches between local and cloud modes, synchronizing the shopping cart status, current stage, conversation history, and most recent valid action during the switch. This allows local mode to seamlessly inherit the conversation state during cloud interruptions, eliminating the need for users to repeat completed ordering steps. Reverse synchronization after network recovery ensures that order data generated during local mode is completely uploaded to the cloud environment, providing a data foundation for order aggregation, data analysis, and cross-device status recovery.

[0094] In some embodiments of the present invention, the system further includes a supervision layer, which includes an anomaly determination rule module and a hierarchical processing module; The anomaly detection rules are used to detect at least one of the following anomalies: number of failed action validations, number of failed field validations, conflict between action and menu constraints, mismatch between action and current stage, price recalculation anomaly, and rollback repair failure. The tiered processing module is configured to: trigger a clarification request when a structured action is determined to be a minor anomaly; trigger a rollback action to restore the structured action when it is determined to be a moderate anomaly; and trigger action rejection, shopping cart state rollback, or session protection interruption when a structured action is determined to be a severe anomaly.

[0095] In this invention, the supervisory layer sits above each functional module, responsible for monitoring the overall system operation and handling exceptions. The exception judgment rule module stores and executes exception detection logic to determine whether a current structured action is an exception. The hierarchical processing module selects different processing strategies based on the severity of the exception. The number of action validation failures refers to the cumulative number of times a structured action fails the legality check of the operation validation module within the same session. The number of field validation failures refers to the cumulative number of times a structured action fails validation due to missing or incorrectly formatted fields within the same session. An action mismatch with the current stage refers to a situation where the legal mapping relationship between the type of the structured action and the current stage of the shopping cart state machine is not satisfied, such as attempting to add a product during the payment stage. A price recalculation exception refers to a situation where, after the order status management module recalculates the price, the sum of the subtotals of each order line is inconsistent with the total amount. A minor exception refers to an exception that has a small impact on order correctness and can be resolved by initiating a clarification request to the user. A moderate exception refers to an exception that has a certain impact on order correctness and can be automatically repaired by the rollback action recovery module. A severe anomaly refers to an anomaly that seriously affects the correctness of an order and cannot be resolved through clarification or rollback. Session protection interruption refers to a protective measure that suspends some or all functions of the current session when a severe anomaly is detected to prevent the error from spreading further.

[0096] In a specific implementation of the present invention, the anomaly judgment rule module of the supervision layer is responsible for detecting various anomaly types, including the number of failed action verifications, the number of failed field verifications, conflicts between actions and menu constraints, mismatches between actions and the current stage, price recalculation anomalies, and rollback repair failures.

[0097] In a specific implementation of this invention, the anomaly determination rule module continuously monitors the status information and data output by each module. Once one or more of the aforementioned anomalies are detected, the anomaly information is sent to the hierarchical processing module. The hierarchical processing module classifies structured actions or system states into three levels—mild, moderate, and severe—based on the severity of the anomaly. For mild anomalies, the hierarchical processing module triggers a clarification request, and the system asks the user for further questions to obtain missing or ambiguous information. For moderate anomalies, the hierarchical processing module triggers a rollback action, and the system attempts to automatically repair the abnormal action. For severe anomalies, the hierarchical processing module triggers action rejection, shopping cart state rollback, or session protection interruption; the system refuses to execute the abnormal action, restores the shopping cart to its previous valid state, or suspends some functions of the current session.

[0098] For example, a user orders food at a self-service terminal. The first exception occurs when the user says "Give me one," without specifying any product name. The structured action output by the dialogue reasoning module lacks a product identifier field. The exception judgment rule module detects a field validation failure and marks the exception as a minor exception. The tiered processing module triggers a clarification request, and the system asks in voice, "What would you like to order?" The second exception occurs when the user says "Change to a large," but the shopping cart only contains a medium-sized Coke. The rollback action recovery module successfully corrects the action to modify the large Coke attribute. The exception judgment rule module detects that the rollback was successful, but the original action is missing, marking the exception as a moderate exception. The tiered processing module triggers the rollback action recovery process, and the action passes validation and is executed. The third exception occurs when the user utters three unrecognizable commands consecutively, and the operation verification module records three failed action validations. The anomaly judgment rule module detected that the number of action verification failures reached the threshold of three times, and the rollback repair also failed twice in a row. The anomaly was marked as a severe anomaly. The graded processing module triggered action rejection, refused to execute the current structured action, triggered the shopping cart state to roll back to the valid snapshot before the three failures, activated session protection interruption, suspended the acceptance of new voice input and prompted the user to restart or contact staff.

[0099] In some embodiments of the present invention, the supervision layer may further include constraint violation classification rules, which at least classify anomalies into categories such as non-existent goods, illegal attributes, missing required attributes, abnormal quantities, price conflicts, and illegal stage transitions. Different rejection, clarification, rollback, or revert strategies can be applied to different categories of anomalies.

[0100] In summary, this invention, by combining a structured action list with a shopping cart state machine, achieves the transition from voice to a deterministic order, ensuring the consistency of the order status in multi-turn dialogues, and improving fault tolerance by reverting to recovery when action verification fails.

[0101] Reference Figure 2 The flowchart illustrates the AI ​​voice-interactive food ordering method, which includes: Step 201: Collect the initial order voice and perform speech activity segmentation on the initial order voice to obtain the target order voice; Step 202: Perform language detection on the target order speech to obtain the target language, and perform standardization processing on the target order speech to obtain standardized text; wherein, the standardization processing includes at least one of noise reduction processing, speech-to-text conversion and text normalization processing; Step 203: Based on standardized text, a preset menu knowledge base, and the current shopping cart status, generate natural language responses and a list of executable structured actions; Step 204: Perform a validity check on each structured action in the structured action list based on the menu knowledge base; Step 205: When a structured action fails validation, repair the structured action based on standardized text, session context, shopping cart snapshot, and menu knowledge base; Step 206: Apply the validated structured actions to the shopping cart state machine, execute constraints, calculate prices, and generate an order summary; Step 207: Use the target language to generate voice feedback information from the natural language response, and output the voice feedback information.

[0102] As the method embodiments are basically similar to the system embodiments, they are described in a relatively simple manner. For relevant details, please refer to the description of the system embodiments.

[0103] This invention also provides an electronic device that may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the method described above.

[0104] This invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the method described above.

[0105] The specific embodiments described above are preferred embodiments of the present invention and are not intended to limit the specific scope of the present invention. The scope of the present invention includes, but is not limited to, these specific embodiments. All equivalent changes made in accordance with the shape and structure of the present invention are within the protection scope of the present invention.

Claims

1. An AI-powered voice-interactive food ordering terminal system, characterized in that: include: The audio interface module is configured to acquire the initial order voice and perform voice activity segmentation on the initial order voice to obtain the target order voice; The speech understanding module is configured to perform language detection on the target order speech to obtain the target language, and to perform standardization processing on the target order speech to obtain standardized text; wherein, the standardization processing includes at least one of noise reduction processing, speech-to-text conversion and text normalization processing; The dialogue reasoning module is configured to generate natural language responses and a list of executable structured actions based on the standardized text, a preset menu knowledge base, and the current shopping cart status. The operation verification module is configured to perform legality verification on each structured action in the structured action list based on the menu knowledge base, and send the structured actions that pass the verification to the order status management module, and send the structured actions that fail the verification to the rollback action recovery module. The rollback action recovery module is configured to repair the structured action based on the standardized text, session context, shopping cart snapshot, and menu knowledge base when the structured action fails to be verified by the operation verification module. The order status management module is configured to apply the validated structured actions to the shopping cart state machine, execute constraints, calculate prices, and generate order summaries. The voice output module is configured to generate voice feedback information from the natural language response in the target language and output the voice feedback information.

2. The system according to claim 1, characterized in that, The speech understanding module includes a speech enhancement unit and a menu terminology standardization unit; The speech enhancement unit is used to improve the speech signal-to-noise ratio in scenarios with background music, device operation sounds, and overlapping speech through noise suppression and target speech separation algorithms. The menu terminology standardization unit is used to map size descriptions, quantity descriptions, and attribute descriptions in user spoken expressions to standard menu codes, so that the dialogue reasoning module can generate the structured action list based on unified codes in multi-turn dialogues; The speech understanding module also includes a speech rate adaptive unit, which is configured to detect the speech rate of the user's speech and adjust the speech enhancement parameters in the speech enhancement unit and the silence detection threshold for speech activity segmentation in the audio interface module according to the detected speech rate, so as to adapt to users with different speech rates.

3. The system according to claim 1, characterized in that, The structured action list is in JSON, key-value pair, or object array format. Each structured action includes at least one of the following fields: action type field, product identifier field, order line identifier field, quantity field, attribute field, confidence field, evidence field, timestamp field, and priority field. The action type field includes at least the following actions: add product, delete product, update product, set attribute, set quantity, apply discount, cancel order, confirm shopping cart, request clarification, and trigger payment. The confidence field characterizes the degree of credibility of the inferred structured action. The evidence field records the statement fragment, reasoning basis, or rule source that triggers the structured action. The priority field determines the execution order when multiple structured actions exist simultaneously.

4. The system according to claim 3, characterized in that, A binding relationship table is established between the structured action list and the shopping cart state machine. The binding relationship table defines the mapping of different action types to state update operations in the shopping cart state machine. Among them, the action of adding a product is mapped to the state transition of adding an order line, the action of deleting a product is mapped to the state transition of removing an order line, the action of setting an attribute is mapped to the state transition of updating an order line attribute, the action of confirming the shopping cart is mapped to the state transition of moving from the ordering stage to the confirmation stage, and the action of triggering payment is mapped to the state transition of moving from the confirmation stage to the payment stage. The binding relationship table further includes conditional mapping rules, which map the same action type to different state transition paths according to the current shopping cart state, wherein the shopping cart state includes at least one of the following: total order amount, product type, or user attribute.

5. The system according to claim 1, characterized in that, The rollback action recovery module is further configured to recover the structured action based on the most recently added product, the most recently modified product, the most recently clarified product, or the currently focused order line when the target product is not explicitly specified in the user statement. The rollback action recovery module is further configured to cancel the structured action of the previous round when the user statement contains a negative expression of the previous round's instruction, and then apply the structured action of the current round.

6. The system according to claim 1, characterized in that, It also includes a supervision layer, which comprises an anomaly detection rule module and a hierarchical processing module; The anomaly determination rules are used to detect at least one of the following anomalies: number of failed action validations, number of failed field validations, conflict between action and menu constraints, mismatch between action and current stage, price recalculation anomaly, and rollback repair failure. The hierarchical processing module is configured to: trigger a clarification request when the structured action is determined to be a minor anomaly; trigger a rollback action to restore when the structured action is determined to be a moderate anomaly; and trigger action rejection, shopping cart state rollback, or session protection interruption when the structured action is determined to be a severe anomaly.

7. The system according to claim 1, characterized in that, The menu knowledge base is a multilingual menu knowledge base, including product definitions, attribute constraints, and business rules, and further includes multilingual mapping data. The multilingual mapping data is used to map product names and attribute values ​​expressed in different languages ​​or mixed languages ​​to a unified internal identifier. Specifically, the same product is mapped to the same internal product identifier in different languages ​​or mixed languages, and the same attribute value is mapped to a unified attribute constraint value in different languages ​​or mixed languages.

8. The system according to claim 1, characterized in that, The system supports dual-mode deployment, including local and cloud modes. In the local mode, the audio interface module, the voice understanding module, the dialogue reasoning module, the operation verification module, the rollback action recovery module, the order status management module, and the voice output module run on the self-service terminal device; In the cloud mode, the self-service terminal device collects the initial ordering voice and transmits it to the server for processing. The initial ordering voice includes at least a session identifier. The local mode and the cloud mode use the same structured action format and the same shopping cart status format.

9. The system according to claim 8, characterized in that, The system also supports switching between the local mode and the cloud mode; When a network interruption or communication quality below a preset threshold is detected, the system switches from cloud mode to local mode and synchronizes the shopping cart status, current stage, dialogue history and the most recent valid action corresponding to the session to the local execution environment. Once the network is restored, the shopping cart changes, action history, and session status during the local mode will be synchronized back to the cloud environment.

10. The system according to claim 1, characterized in that, It also includes a session state storage module, which is configured to generate and save a shopping cart state snapshot and a dialogue context vector in each round of dialogue; The rollback action recovery module is further configured to, when the structured action verification fails, call the most recent valid shopping cart state snapshot and reconstruct the user intent in conjunction with the dialogue context vector to generate a corrected structured action.

Images