Real-time voice order placing method and system based on multi-modal artificial intelligence

Abstract

The application relates to the technical field of artificial intelligence, and discloses a real-time voice order placing method and system based on multi-modal artificial intelligence, aiming to solve the problems of poor robustness, high delay and weak generalization capability of existing voice order placing technology in complex environments. The method comprises the following steps: collecting user voice and synchronously obtaining text history; performing feature extraction on multi-modal data and realizing semantic alignment through a cross-modal attention mechanism; generating a fused intention representation by dynamically weighting through a gated fusion network, and outputting a structured task label through a hierarchical intention classifier; calling a task template engine to generate a compliant order; distributing the order through a high-priority message queue with low delay and constructing a feedback loop to support online model optimization. The above scheme realizes the unification of high robustness, low delay and strong generalization capability, and significantly improves the accuracy and real-time performance of voice order placing in noisy, dialectal and high-concurrency scenarios.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to a real-time voice order delivery method and system based on multimodal artificial intelligence. Background Technology

[0002] Voice order placement technology, as a key human-computer interaction method that transforms natural speech commands into structured task orders, is widely used in fields such as intelligent scheduling and instant services. However, existing technical solutions face significant limitations in complex real-world scenarios.

[0003] First, most systems rely on a single speech modality for recognition and understanding. In the presence of noise interference, user accents, or semantic ambiguity, the accuracy and robustness of intent recognition drop sharply, making it difficult to adapt to real dynamic environments.

[0004] Secondly, traditional systems typically employ an architecture where modules such as speech recognition, semantic understanding, and order generation are executed sequentially, resulting in long processing links and high cumulative latency, which cannot meet the real-time business requirements of high concurrency and low latency.

[0005] Furthermore, although some solutions attempt to incorporate multimodal information such as text and context, they lack effective cross-modal semantic alignment and dynamic fusion mechanisms. The information of each modality is often processed in isolation, making it difficult to achieve synergistic enhancement. When the quality of some modalities deteriorates, the system performance becomes unstable.

[0006] Finally, existing systems have limited generalization capabilities when faced with diverse user expressions, such as dialects, industry terms, or non-standard spoken language, and they usually lack the ability to continuously self-optimize, resulting in performance stagnation during long-term use and difficulty in adapting to new scenarios and expressions.

[0007] Therefore, there is an urgent need for a voice order delivery solution that can integrate multimodal information, achieve efficient collaborative processing, and possess strong robustness and continuous learning capabilities to solve the above problems. Summary of the Invention

[0008] The purpose of this invention is to provide a real-time voice order delivery method and system based on multimodal artificial intelligence, which can effectively solve the problems in the background art mentioned above.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A real-time voice order placement method based on multimodal artificial intelligence includes the following specific steps: In response to the user's voice order command, the system synchronously collects and time-aligns the user's voice, historical interaction text, and environmental awareness data to form a multimodal input. Parallel feature extraction is performed on the multimodal input, and the alignment of speech, text and environmental features in the semantic space is achieved through a cross-modal attention mechanism to generate a unified aligned feature representation. The aligned feature representation is input into a gated fusion network. The gated fusion network dynamically generates fusion weights based on the input quality of each modality, performs weighted fusion on the aligned features, and inputs the weighted fused features into a hierarchical intent classifier to output structured task intent labels. The corresponding structured task template is invoked based on the task intent tag, and key entity information is extracted based on the aligned feature representation to fill the task template, thereby generating a structured task order. The task orders are distributed with low latency through a high-priority in-memory message queue, and order execution feedback is collected synchronously. An online optimization loop is built based on the feedback to continuously optimize the performance of the gating fusion network and the hierarchical intent classifier.

[0010] Furthermore, user voice, historical interaction text, and environmental awareness data are collected and time-aligned simultaneously, including: The raw audio stream is acquired using a multi-channel microphone array at a preset sampling rate and divided into audio frames of fixed duration. Synchronously query the local interaction database to obtain historical interaction text records for a preset number of rounds, forming a text sequence that represents the current dialogue context; Environmental perception data is read from integrated multiple sensors at a fixed frequency, and the environmental perception data includes at least light intensity, geographical location and motion status data; The audio frames, the text sequences, and the environmental awareness data are each stamped with a high-precision timestamp provided by a precision clock synchronization protocol, and data is exchanged through shared memory to complete time alignment.

[0011] Furthermore, parallel feature extraction is performed on the multimodal inputs, including: After pre-emphasis, windowing, and short-time Fourier transform of the audio frame, the Mel frequency cepstral coefficients and filter bank energy characteristics are calculated and spliced ​​to form an acoustic feature vector. The text sequence is input into a pre-trained and fine-tuned semantic encoding model, which outputs a text semantic vector. After normalizing and organizing the time-series signals reported by environmental sensors, the signals are input into a one-dimensional convolutional neural network for feature extraction, and the output is an environmental context feature vector.

[0012] Furthermore, semantic space alignment is achieved through a cross-modal attention mechanism, specifically as follows: Using the acoustic feature vector sequence as the query and the text semantic vector as the key and value, calculate the cross-modal attention weights; The cross-modal attention weights are used to perform a weighted summation of the environmental context feature vector sequence, guiding environmental information to focus on the key semantics of the text description; The attention-weighted acoustic features, textual semantic features, and environmental features are concatenated along the feature dimension and then processed by layer normalization to generate the unified aligned feature representation.

[0013] Furthermore, the gated fusion network dynamically generates fusion weights based on the input quality of each modality, including: Collect feature indicators that reflect the input quality of each modality, including at least the real-time signal-to-noise ratio of speech, the length of the current text sequence, and the variance of environmental sensor data; The unified aligned feature representation and the feature index are input together into a modality confidence evaluation subnetwork. This subnetwork dynamically outputs three weight coefficients corresponding to speech, text and environment modalities. The weight coefficients satisfy normalization constraints and are within a preset empirical value range.

[0014] Furthermore, the hierarchical intent classifier employs a two-layer structure: The first layer is the Softmax classification layer, which is used to categorize user intents into preset top-level business categories; The second layer is a refinement classifier corresponding to the top-level business category, which is used to further identify specific sub-intents under the top-level business category and finally output a one-hot encoded vector corresponding to the structured task intent label.

[0015] Furthermore, the task template is populated by extracting key entity information based on the aligned feature representation, including: The text semantic vector is input into a named entity recognition model specifically for order generation to identify and extract entity values ​​of location, item, and quantity from the text context; Real-time keyword detection is performed on the acoustic feature sequence, and standardized item or attribute identifiers are obtained by matching a predefined keyword pronunciation feature library. According to the predefined field-entity type mapping rules, the extracted entity values ​​and the standardized identifiers are filled into the corresponding fields of the task template, and the unfilled fields are filled using historical default values.

[0016] Furthermore, low-latency distribution is achieved through a high-priority in-memory message queue, including: The structured task order is serialized into a compact binary byte stream; The binary byte stream is submitted to a lock-free high-priority message queue implemented based on a shared memory circular buffer; The message queue server thread sends the binary byte stream to the downstream task execution system in real time through a maintained TCP long connection using zero-copy technology.

[0017] Furthermore, an online optimization closed loop is constructed, including: Order execution feedback is collected from three dimensions: explicit user confirmation, downstream execution system feedback, and implicit signals automatically identified by the system. The feedback, along with the corresponding order data and multimodal feature vectors, is encapsulated into a feedback record and stored. Based on the feedback records periodically, a reinforcement learning algorithm is used to incrementally fine-tune the parameters of the gating fusion network and the hierarchical intent classifier, using task success rate and user confirmation rate as reward signals.

[0018] On the other hand, the real-time voice order system based on multimodal artificial intelligence disclosed in this application includes: The multimodal data acquisition and preprocessing module is used to respond to voice order instructions, simultaneously acquire user voice, historical interactive text and environmental awareness data, and perform time alignment. The cross-modal feature extraction and alignment module is used to extract features from speech, text and environment in parallel, and to achieve semantic-level feature alignment through a cross-modal attention mechanism to generate a unified aligned feature representation. The multimodal fusion intent recognition module includes a gated fusion network and a hierarchical intent classifier. The gated fusion network is used to dynamically weight and fuse the aligned features based on the input quality. The hierarchical intent classifier is used to classify the weighted fusion features and output a structured task intent label. The structured task order generation module is used to call the task template according to the task intent tag, extract entity information from the aligned feature representation to fill the template, and generate a structured task order that passes the business logic verification. The low-latency distribution and online optimization module is used to distribute the task orders through a high-priority message queue, collect execution feedback, and drive the online optimization of the fusion intent recognition module based on the feedback data.

[0019] In summary, this application includes at least one of the following beneficial technical effects: 1. This application overcomes the limitations of single-modal speech recognition: by fusing speech, text context, and environmental perception signals, a unified semantic understanding framework is constructed. In scenarios with strong noise, accent interference, or semantic ambiguity, the intent recognition accuracy reaches a preset accuracy threshold, which is more than a preset percentage point higher than the traditional single-modal solution. Dynamic modality weight adjustment mechanism: the gated fusion network adaptively adjusts the contribution of each modality according to the input quality, ensuring that high-precision parsing can still be maintained by relying on historical interactions and environmental cues when the speech signal deteriorates, significantly improving the stability of the system in complex dynamic environments.

[0020] 2. Parallelized multimodal processing pipeline of this application: Abandoning the traditional serial architecture, it realizes parallel execution of voice acquisition, feature extraction, fusion recognition and order generation pipeline, and compresses the end-to-end latency to within the preset latency threshold to meet the demanding real-time requirements of logistics scheduling, emergency command and other scenarios; Edge intelligent deployment mode: The whole process is completed at the edge node, avoiding round-trip transmission to the cloud, and a single node supports a preset number of concurrent requests per second, and the system throughput is improved by more than a preset multiple compared with the centralized architecture.

[0021] 3. The technical solution of this application covers all industry scenarios: It has a built-in number of standardized task templates, which can be quickly adapted to new business fields without retraining the core model; The online self-optimization mechanism: Based on the reinforcement learning closed loop of user feedback, the system continuously improves its understanding of dialects, terminology and non-standard expressions in actual operation. The periodic accuracy improvement mechanism ensures that the long-term performance does not decay, which solves the problem of the solidification of the generalization ability of the existing system. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the architecture of a real-time voice order delivery method based on multimodal artificial intelligence; Figure 2 This is a schematic diagram illustrating the core principle framework of a multimodal fusion intent recognition model; Figure 3 This is a flowchart illustrating the process framework for multimodal data acquisition, feature alignment, and task order generation. Detailed Implementation

[0023] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0024] Example 1 This embodiment is applied to a real-time voice order placement scenario in a city-level intelligent logistics dispatch center. In this scenario, dispatchers issue highly time-sensitive voice commands such as "urgently transfer 50 boxes of epidemic prevention materials from Warehouse 3 in Area A to the east gate of Hospital B" through wearable terminal devices. The system needs to complete intent recognition, order generation, and task distribution within 150 milliseconds, and ensure that it can maintain an intent recognition accuracy rate of over 98.5% even in a complex acoustic environment with background noise of up to 60 decibels and interference from multiple people talking at the same time.

[0025] Firstly, this application discloses a real-time voice order system based on multimodal artificial intelligence. A hardware and software collaborative platform is constructed at the system architecture level to support the aforementioned highly robust and low-latency interaction capabilities. This system is deployed across distributed edge computing nodes, each of which is an independent processing unit integrating computing, sensing, and communication functions. Its architecture is as follows: 1. Hardware platform composition; (1) Core processing unit: Each edge node is equipped with an edge AI computing module with neural network inference acceleration capability, which is used to carry out localized real-time inference tasks throughout the entire process.

[0026] (2) Voice acquisition and enhancement subsystem: Multi-channel microphone array: An omnidirectional microphone array with a ring layout is used to acquire audio at a preset sampling rate.

[0027] Noise suppression processing: The array integrates a beamforming coprocessor to run an adaptive noise suppression algorithm, improving speech clarity in complex acoustic environments.

[0028] (3) Multi-source context-aware subsystem: Interaction history module: caches recent human-computer interaction logs through a local database, including text transcription and task context.

[0029] Environmental sensing module: Integrates multiple sensors to collect environmental information such as lighting, position, and motion. Sensor data is preprocessed and time-synchronized by the microcontroller before being transmitted to the main processor via a high-speed bus.

[0030] (4) Communication and distribution subsystem: Network interface: Equipped with a highly reliable wired network interface, supporting a precision clock synchronization protocol to ensure low jitter when connected to downstream systems.

[0031] Message queue: A high-priority order queue is implemented using an in-memory-resident message middleware, supporting high concurrency and low latency message distribution.

[0032] Real-time performance guarantee: The operating system adopts a real-time scheduling strategy to prioritize critical task threads and bind them to CPU cores to ensure processing timeliness.

[0033] 2. System Architecture; The system architecture is based on a modular design and works closely with the hardware. It mainly includes the following functional modules: Multimodal data acquisition and preprocessing module: responsible for synchronously acquiring voice, text history and environmental sensor data, and performing time alignment and preliminary cleaning.

[0034] Cross-modal feature extraction and alignment module: Extracts features of each modality in parallel and achieves semantic-level cross-modal alignment through an attention mechanism.

[0035] Multimodal fusion intent recognition module: It uses a gated fusion network to dynamically integrate multimodal information and outputs structured intent labels through a hierarchical classifier.

[0036] Structured task order generation module: Based on intent tags, it calls task templates, extracts entity information, and generates compliant orders.

[0037] Low-latency distribution and online optimization module: manages order distribution, collects feedback, and supports continuous model optimization.

[0038] 3. System Collaborative Working Mechanism: Hardware and the system collaborate through driver layer and system calls: Sensor data is acquired by the driver and then sent to the software pipeline; computationally intensive tasks such as feature extraction and model inference are accelerated by dedicated hardware; generated orders are sent out in real time through high-speed message queues and network interfaces. Each software module executes in parallel in a pipelined manner, uniformly scheduled by a real-time operating system to ensure that end-to-end processing latency meets stringent requirements.

[0039] On the other hand, the real-time voice order-issuing method based on multimodal artificial intelligence disclosed in this application includes the following steps: The first step is S1: multimodal data acquisition and synchronization; when the user triggers a voice order command through a physical button or a specific wake word, the system immediately starts the multimodal data acquisition and preprocessing process.

[0040] S101: The system concurrently activates three independent data acquisition channels: audio acquisition channel, text history channel, and environmental awareness channel.

[0041] S102: The audio acquisition channel drives the microphone array to continuously capture raw audio at a sampling rate of 48 kHz and quantize it into a continuous data stream in a 16-bit deep pulse-code modulation format. This data stream is then divided into consecutive frame units of 20 milliseconds each, with each frame containing 960 sample points, and fed into a pre-allocated circular buffer for further processing.

[0042] S103: The text history channel synchronously accesses the local lightweight database to query and read the most recent 20 rounds of human-computer interaction records. These records contain at least a timestamp, speech-to-text transcription, and the task order identifier generated in the previous round. The system concatenates these records in chronological order to form a coherent text sequence with a total length not exceeding 512 characters, which is used to represent the current dialogue context.

[0043] S104: The environmental sensing channel reads real-time physical information from integrated multiple sensors at a fixed frequency of 100 Hz. This information includes at least: ambient light intensity in lux, geodetic latitude and longitude and altitude based on the WGS-84 coordinate system, and linear acceleration and angular velocity data of the terminal device in three-dimensional space.

[0044] S105: To ensure the accuracy of subsequent fusion processing, the system tags each data unit acquired from the three channels with a timestamp tag with nanosecond-level accuracy provided by a precision clock synchronization protocol. All tagged data is transmitted to the next stage processing module through the shared memory area, a high-speed data exchange region.

[0045] Step S1 completes the synchronous acquisition and alignment of multimodal raw data. Its technical purpose is to provide temporally synchronized and semantically interconnected speech, text, and environmental context information for subsequent intent understanding, forming a reliable data foundation for the entire system's high-precision understanding. The end-to-end processing time in this stage is strictly controlled within 5 milliseconds, with latency primarily limited by the direct memory access transmission mechanism of the audio interface and the arbitration mechanism of the flexible data bus in the control area network. At this point, the system has completed the construction of a high-quality multimodal data foundation, preparing for subsequent feature extraction and semantic alignment.

[0046] Regarding step S2: the multimodal feature extraction and alignment step; after the data enters the cross-modal feature extraction and alignment module, the system performs parallel deep feature extraction and semantic space mapping on the three streams of data: speech, text, and environment, including the following sub-steps: S201: First, the original pulse code modulation audio frame is pre-emphasized using a pre-emphasis coefficient of 0.97. Then, the frame is divided into consecutive overlapping windows with a duration of 25 milliseconds and a step size of 10 milliseconds. For each window, its short-time spectrum is calculated and passed through a set of Mel-scale triangular filters covering the frequency range of human hearing.

[0047] The output energy of each filter in the filter bank is calculated to obtain the energy characteristics of the filter bank. Simultaneously, the natural logarithm of these energy values ​​is taken, followed by a discrete cosine transform, and the resulting set of coefficients is selected as the Mel-frequency cepstral coefficient characteristics.

[0048] In this embodiment, the calculated Mel-frequency cepstral coefficients are 128-dimensional, and the filter bank energy features are 40-dimensional. Finally, these two sets of feature vectors are concatenated along their feature dimensions to form a 128-dimensional acoustic feature vector sequence. This feature calculation process is completed using the parallel computing core of the graphics processing unit and utilizes the Fast Fourier Transform library to accelerate the Short-Time Fourier Transform operation. Feature extraction for a single frame of speech takes approximately 3 milliseconds.

[0049] S202: The concatenated context text sequence is input into a semantic encoding model for encoding. This model is based on the Transformer architecture and has been pre-trained on a massive Chinese corpus. To adapt to the voice order delivery domain, the model is supervisedly fine-tuned using domain corpus containing task instructions and order annotations. Fine-tuning tasks include masked language modeling and next-sentence prediction.

[0050] The fine-tuned model parameters are solidified and deployed in a dedicated neural network inference engine to achieve low-latency forward inference. The encoder encodes the input sequence and outputs a 768-dimensional dense semantic vector.

[0051] To balance semantic understanding depth with system latency, the system employs a hierarchical encoding strategy: the last speech-to-text message from the current user is fully context-encoded, while historical interaction records are masked using an attention mechanism, allowing them to participate only as background information in the generation of the current semantic vector, without undergoing full recursive encoding. The inference time for this text semantic encoding step is approximately 8 milliseconds.

[0052] S203: Perform normalization preprocessing on the raw signals reported by environmental sensors. Linearly map the light intensity to a closed interval of 0 to 1. After converting the geographic coordinates to planar coordinates via Mercator projection, scale them to bring them into a similar numerical range.

[0053] The acceleration readings were divided by the standard gravitational acceleration constant and converted to gravitational acceleration g as the unit. These normalized data were then organized chronologically to form a time-series window of length 50, covering continuous historical data within 500 milliseconds prior to the current system moment.

[0054] The data from this time window is fed into a one-dimensional convolutional neural network for feature extraction. The network contains three cascaded convolutional layers with kernel sizes of 7, 5, and 3, respectively, and the number of feature channels output by each layer is 64, 128, and 256.

[0055] Each convolutional layer is followed by batch normalization and linear rectified function activation. The network finally outputs a 128-dimensional context feature vector through a global pooling layer and a fully connected layer. This temporal modeling process takes approximately 4 milliseconds.

[0056] The steps S201 to S203 described above achieve parallel feature extraction of trimodal data. The technical objective is to transform the raw, heterogeneous speech, text, and environmental data into high-level, fusion-ready numerical feature representations, laying the foundation for subsequent cross-modal semantic alignment.

[0057] S204: After obtaining the three-way features, the system activates the cross-modal attention alignment unit. This unit employs a multi-head cross-attention mechanism, and its core operation is defined by the following formula: In this formula, The query matrix is ​​obtained by transforming the sequence of speech acoustic feature vectors generated in step S201 through a trainable linear projection layer. The key matrix is ​​obtained by transforming the text semantic vector generated in step S202 through another trainable linear projection layer. The representative value matrix is ​​also obtained by transforming the text semantic vector through a third trainable linear projection layer. The dimension representing the key vector is an adjustable hyperparameter used to scale the dot product result and prevent gradient vanishing.

[0058] Specifically, this mechanism uses speech features as queries and text features as keys and values. By calculating the similarity weights between speech and text, an attention distribution is obtained. This attention distribution is then used to perform a weighted summation of the environmental feature vector sequence, thereby dynamically guiding environmental information to focus on key semantic segments in the text description.

[0059] For example, when the text semantic vector encodes the location information "East Gate of Hospital B", the calculated attention weights will enhance the environmental localization features related to concepts such as "hospital" and "east gate".

[0060] S205: After cross-modal attention alignment, features from speech, text, and environment are directly concatenated along the feature dimension to form a unified feature representation whose dimension is the sum of the feature dimensions of the three modalities. This fused feature is then processed through a layer normalization layer to stabilize its numerical distribution and accelerate the convergence of subsequent models.

[0061] The total time for the entire feature extraction and alignment stage is approximately 12 milliseconds, and its performance bottleneck mainly lies in the bandwidth of the graphics processor memory.

[0062] In summary, step S2 completes the transformation from multimodal raw data to aligned, high-dimensional semantic features. Through a parallel dedicated feature extractor and an attention-based semantic alignment mechanism, the system effectively resolves the semantic gaps between different modalities, generating a unified representation that comprehensively reflects user commands, dialogue history, and physical context. This high-quality feature representation provides direct and reliable input for the next step of accurate dynamic fusion and intent recognition.

[0063] Regarding step S3: the multimodal fusion intent recognition step; after obtaining the aligned fusion feature representation, the system enters the multimodal fusion intent recognition stage. This stage aims to accurately output structured user intent labels through dynamic fusion strategies and hierarchical classification, and includes the following sub-steps: S301: The system first inputs the unified feature representation output from step S2 into a lightweight modal confidence evaluation subnetwork.

[0064] This subnetwork is a two-layer fully connected neural network, with the first layer containing 256 neurons and the second layer containing 128 neurons, and uses a linear rectified function as the activation function.

[0065] Meanwhile, the system collects feature indicators reflecting the input quality of each modality as auxiliary inputs for this sub-network. These indicators include at least: the real-time speech signal-to-noise ratio estimate obtained from the speech front-end processing module, the length of the current text sequence, and the variance of environmental sensor data within the recent time window.

[0066] The evaluation subnetwork takes the aforementioned fusion features and quality indicators as input, and dynamically outputs three weight coefficients corresponding to speech, text, and environmental modalities through forward propagation calculation, denoted as follows: , and .

[0067] These three weighting coefficients satisfy the normalization constraint, that is... .

[0068] Meanwhile, the system sets adjustable empirical value ranges as constraints for each weight, including the voice weight. The value ranges from 0.3 to 0.7, indicating the text weight. The value ranges from 0.2 to 0.5, representing the environmental weight. The value ranges from 0.1 to 0.3.

[0069] The flexibility of this mechanism lies in the fact that the subnetwork can dynamically adjust the weight allocation based on the input quality. For example, when the evaluation subnetwork detects that the speech signal-to-noise ratio is below 50 dB, it may adjust the speech weights. Reduce it to 0.4, and correspondingly adjust the text weight. The value was increased to 0.45 to reduce the impact of poor speech signals and rely more on reliable text context.

[0070] S302: After obtaining the dynamic weights, the system performs weighted fusion on the modal-aligned features from step S204 that have not yet undergone final splicing. Specifically, the speech feature vector is multiplied by the weights. Text feature vectors multiplied by weights Environmental feature vector multiplied by weights Then, the three weighted feature vectors are concatenated along the feature dimension to form the final weighted fusion feature.

[0071] The weighted fusion feature is then fed into a hierarchical intent classifier for classification decision.

[0072] This classifier uses a two-layer structure to achieve coarse-to-fine intent recognition.

[0073] The first layer is a Softmax classification layer with 5 nodes, which is responsible for classifying user intents into top-level categories. For example, the current instruction can be divided into 5 preset main business categories such as "logistics scheduling", "food delivery", and "emergency response".

[0074] The class probabilities output by the first layer will activate the corresponding second-layer refinement classifier. Each second-layer classifier is a multi-class classifier for that broad category, responsible for further identifying specific sub-intents.

[0075] For example, if the first layer is determined to be the "logistics scheduling" category, the logistics scheduling subclassifier is activated. This subclassifier can then select from 40 preset logistics sub-intentions, such as "emergency medical supplies scheduling" and "general warehousing allocation".

[0076] Finally, the hierarchical classifier outputs a 200-dimensional one-hot encoded vector, which uniquely corresponds to a structured task intent label.

[0077] In summary, step S3 completes the crucial transformation from multimodal fusion features to structured intent labels. By introducing a dynamic weighted fusion mechanism based on input quality, the system significantly improves its robustness to understanding under adverse conditions such as damaged speech signals. The hierarchical classification architecture ensures precise recognition of intents across a wide range of business scenarios. The accurate intent labels output in this step provide a direct basis for subsequently generating clearly structured and field-defined task orders. The entire fusion and classification process, accelerated by a dedicated inference engine, takes approximately 15 milliseconds, further improving accuracy.

[0078] Step S4: Structured Task Order Generation; After obtaining the structured intent tags, the system enters the task order generation stage. The goal of this stage is to transform the abstract intent into a specific, executable, and business-compliant structured order, which includes the following sub-steps: S401: The system first retrieves a matching order template from the pre-set task template library based on the intent tag output in step S3. This template library is stored in the system's non-volatile memory, and a mapping relationship is established between the intent tag and the corresponding JSON structure template file path in the form of key-value pairs.

[0079] For example, when the intent label is "emergency medical supplies dispatch", the system locates the corresponding logistics dispatch template file path by querying the mapping table and loads its content.

[0080] This template library includes 12 types of structured data format definition files that conform to industry-standard specifications. Each template defines the set of fields required to generate that type of order, the data type of each field, and the value range constraints.

[0081] Taking logistics dispatch intent as an example, the matching template explicitly defines four required fields: source warehouse, destination, goods type, and urgency. The system then creates a corresponding order object instance in memory with empty values ​​for each field based on the loaded JSON structure, serving as the framework for generating this order.

[0082] S402: Next, based on the aligned multimodal features generated in step S2, the system initiates a parallel information extraction process to fill in the various fields in the template.

[0083] First, the system inputs the text semantic vectors generated in step S202, which encode the current and historical dialogue context, into a named entity recognition (NER) model specifically fine-tuned for the order generation task. This NER model, trained on a large corpus of text labeled with order-related entities such as location, item, quantity, and time, has the ability to identify and classify these entities from the context semantic vectors.

[0084] Specifically, the model decodes the semantic vector and outputs a sequence labeling result, indicating the entity type corresponding to each position in the text. Based on the labeling result, the system extracts the corresponding text fragments from the original text sequence as entity values.

[0085] For example, the model identifies and extracts the string "East Gate of Hospital B" from a text sequence and marks it as a "Destination" entity.

[0086] Simultaneously, the system performs real-time keyword detection on the acoustic feature sequence extracted in step S201. The system maintains a predefined keyword pronunciation feature library containing common items, actions, and attributes within the domain. Each keyword in this feature library is associated with a standardized internal identifier.

[0087] The system uses algorithms such as dynamic time warping to calculate the similarity between real-time acoustic feature sequences and each keyword template in the feature library. When the similarity exceeds a preset threshold, the system determines that the keyword has been spoken. For example, when the pronunciation feature of "epidemic prevention supplies" is detected to be matched, the system obtains the standard ID of the goods type associated with that keyword.

[0088] S403: The system, based on predefined field-entity type mapping rules, fills the entity values ​​extracted in step S402 into the blank order object instance created in S401. These mapping rules specify which specific field of the order object each type of entity value should be filled into.

[0089] For example, the mapping rules stipulate that entity values ​​of type "destination" should be filled into the destination field of the order object, while standard cargo type IDs obtained from voice keyword mapping should be filled into the cargo_type field.

[0090] The system performs a population operation. For fields defined in the mapping rules but not directly extracted from the current multimodal information, the system will query the interaction history database for the corresponding value from the most recent task with the same intent and use it as the default value for population.

[0091] S404: After all fields of the order object are populated, the system calls the constraint validator to perform a series of preset business logic checks. These checks are implemented by calling external service interfaces or querying the local database.

[0092] The first check is service area verification. The system converts the destination text description in the order object into latitude and longitude coordinates using a geocoding service, and then queries the predefined service area polygon geofence data to determine whether the coordinate point is located inside any service fence polygon. If it is located outside, it is determined to be out of bounds.

[0093] The second check is inventory and resource matching verification. Based on the source warehouse field and goods type field in the order object, the system constructs a query statement, accesses the real-time inventory database interface of the warehouse management system, and checks whether there is a sufficient quantity of the specified type of goods in the specified warehouse.

[0094] S405: The constraint validator returns the result of each check. If all checks pass, the order object is marked as compliant, and the process proceeds to the next distribution stage.

[0095] If any check fails, the validator returns a structured error object containing the specific failure reason code and details. The system pre-configures a corresponding natural language clarification script template for each possible failure reason code. Based on the reason code in the received error object, the system selects the corresponding script template and fills in the key variables from the error details to generate a complete clarification query.

[0096] For example, if service area verification fails and the error details include the unrecognizable address text "East Gate of Hospital B", the system might select a template like "Your specified destination '{address}' is not within the service area. Please reconfirm or provide nearby landmarks." The system then plays the inquiry through a speech synthesis module or displays the text through a graphical interface, and enters a listening state to collect supplementary information while waiting for the user's voice response.

[0097] In summary, step S4 successfully instantiates the identified user intent into a well-structured, complete, and business-validated executable order. It achieves efficient and accurate order generation through the organic combination of templating, multimodal information extraction, and rule validation. The compliant orders generated in this step lay a solid foundation for ultimately achieving millisecond-level low-latency and reliable distribution.

[0098] Regarding step S5: Low-latency distribution and feedback loop; after generating compliant structured orders, the system enters a closed-loop stage of order distribution and continuous optimization. This stage aims to achieve extremely fast and reliable order distribution, and simultaneously collect execution feedback to drive the self-evolution of the system model. Specifically, it includes the following sub-steps: S501: The system first serializes the structured order object generated in memory in step S4. This process calls the serialization code compiled from the Protocol Buffers definition file to encode each field of the order object according to a predefined binary format, generating a compact binary byte stream. This encoding method significantly reduces the amount of data transmitted over the network and facilitates efficient subsequent parsing.

[0099] S502: The serialized order byte stream is immediately submitted to a high-priority message queue. This queue is implemented based on shared memory, and its core is a lock-free circular buffer with pre-allocated fixed-capacity memory blocks. The submission operation ensures thread safety through atomic instructions, and the order data is directly written to the next available position in the buffer. Because the entire operation is completed in memory and lock contention is avoided, the average latency of enqueuing a single order can be controlled to within 0.5 milliseconds.

[0100] S503: The message queue's server thread continuously monitors the buffer. Once a new order data write is detected, it immediately sends the order's binary stream to the downstream task execution system via an established TCP long connection. A keep-alive mechanism is maintained between the system and the downstream execution system, employing zero-copy network transmission technology to further reduce processing overhead and transmission latency. The entire end-to-end network transmission latency, from order generation in the queue to confirmation of receipt by the downstream system, has been verified to be less than 20 milliseconds.

[0101] S504: Simultaneously with order placement, the system initiates an asynchronous feedback listening process. This process collects feedback signals related to the execution of this order from three dimensions in parallel.

[0102] The first dimension is the user's explicit confirmation. The system captures the user's confirmatory phrases such as "received" and "okay" through voice endpoint detection or graphical interface interaction.

[0103] The second dimension is a structured receipt from the downstream execution system, which is returned through a separate communication link and contains fields such as task status, completion time, or reason for failure.

[0104] The third dimension consists of implicit signals that the system can autonomously identify. For example, the system may monitor whether a user repeatedly issues the same or similar commands within a very short time window after an order is placed, such as within 2 seconds. If such behavior is detected, the system can consider it implicit negative feedback that the order processing did not meet the user's needs.

[0105] S505: The collected raw feedback signals are sent to the feature engineering module for structured processing. This module calculates a series of quantitative indicators, such as the total time taken from task issuance to receiving a completion receipt, the confirmation delay from when the user gives a command to when the user gives an explicit confirmation, and a Boolean flag indicating whether the task was successful or not based on the receipt status.

[0106] The calculated feature metrics, along with the corresponding original order data and the multimodal feature vector used to generate the order, are collectively encapsulated into a complete feedback record. This record is then written into a dedicated feedback database, forming historical data that can be used for subsequent analysis.

[0107] In summary, step S5 completes the entire business and technology loop from order generation to delivery and data retrieval. It not only ensures the instantaneous availability of core business instructions through memory queues and optimized network links, but more importantly, it builds a sustainable learning feedback ecosystem. The high-quality interactive data accumulated in this stage directly serves the online model adaptive optimization mechanism, which will be detailed below. This allows the system to break free from the limitations of static models, continuously evolve in actual operation, and ultimately achieve steady long-term performance improvement and excellent generalization capabilities.

[0108] An online adaptive optimization mechanism is implemented; the system initiates incremental training every Sunday at 2 AM. The reinforcement learning agent uses task success rate and user confirmation rate as reward signals, and employs the PPO algorithm to fine-tune the parameters of the last two layers of the gating fusion network and the intent classifier. Training data comes from 100,000 real interaction samples from the past 7 days, which are processed for differential privacy and then used for gradient updates. Experimental results show that this mechanism steadily improves the system's accuracy in recognizing dialects (such as Cantonese and Sichuanese) and industry terms (such as "cold chain BC category"), effectively addressing the problem of model performance degradation.

[0109] Example 2 This embodiment is deployed in an urban fire command center, addressing the voice-based dispatching needs of frontline firefighters. A typical command is, "Requesting reinforcements of 2 aerial ladder trucks and 1 chemical decontamination team, coordinates 39.9042°N, 116.4074°E." This scenario presents three unique challenges: extreme noise (>80 dB), violent equipment movement (acceleration > 3g), and intermittent network outages. Therefore, the system has been specifically enhanced based on Embodiment 1.

[0110] On the hardware side, the microphone array has been upgraded to a 12-channel directional microphone with a 100 dB SPL overload point. Combined with a self-developed anti-saturation preamplifier, the dynamic range has been extended to 95 dB. The environmental perception subsystem adds a Bosch BME680 quad-sensor to determine the fire hazard level. The communication module adds a LoRaWAN backup link to cache orders to FRAM non-volatile memory during network outages.

[0111] At the software level, the task template engine loads a dedicated "emergency response" schema, including fields such as hazard_type (fire / chemical leak, etc.) and required_resources (encoded according to NFPA standards). The online optimization module introduces a few-shot learning mechanism, utilizing a fire terminology dictionary (containing 5000+ professional terms) for fine-tuning of the embedding layer. The implementation steps are as follows: In the data acquisition phase of step S1, a motion state detection module was added to the system. This module receives and processes the raw data stream from the six-axis inertial measurement unit in real time.

[0112] The system continuously calculates the synthetic acceleration amplitude of the device in three-dimensional space. When the synthetic acceleration amplitude is detected to continuously exceed 3 times the standard gravitational acceleration and the state is maintained for 100 milliseconds, the system determines that the device is in a state of violent motion and automatically switches the voice acquisition front end to a high interference immunity mode.

[0113] In this mode, the system dynamically adjusts two key parameters: First, it temporarily increases the sampling rate of the audio acquisition channel from the baseline to 96kHz to capture richer acoustic details to combat motion noise; second, it controls the beamforming coprocessor to switch to a super-directional algorithm to narrow the main lobe width of the acoustic receiver to within 30 degrees, thereby focusing more accurately on the direction of the target sound source and suppressing interference from other directions.

[0114] In step S2, environmental temporal feature modeling, the system expands the perception dimensions. In addition to the existing illumination, position, and motion information, temporal signals of volatile organic compound gas concentration, temperature, and air pressure are newly acquired. These new environmental signals, after undergoing the same normalization and time window organization, are fed into an enhanced one-dimensional convolutional neural network for feature extraction. The output dimension of the last fully connected layer of this network is adjusted to 192 dimensions to accommodate richer environmental context information.

[0115] Simultaneously, prior to cross-modal attention alignment, the system introduces a motion compensation preprocessing step. Specifically, the system utilizes high-frequency acceleration data acquired from the inertial measurement unit to analyze the microscopic motion trajectory of the device during speech acquisition. Based on this trajectory, a dynamic time warping algorithm is employed to perform nonlinear alignment and correction on the time axis of the acoustic feature vector sequence generated in step S201. This counteracts temporal distortion and aberration of the speech signal caused by severe shaking or impact of the device, thereby improving the robustness of subsequent speech features under motion interference.

[0116] In the multimodal fusion intent recognition step S3, the system adjusts the constraint boundaries of the modal weighting strategy based on the characteristics of the fire emergency scenario. Since gas concentration and temperature data play a crucial indicative role in assessing the fire situation, the system weights the environmental modal coefficients accordingly. The adjustable upper limit is increased from 0.3 for general scenarios to 0.4. Correspondingly, speech reliability may decrease in extreme noise environments, therefore the speech modality weighting coefficient is adjusted. The adjustable lower limit is reduced from 0.3 to 0.25, making the weight allocation mechanism more flexible to adapt to the fire environment.

[0117] Furthermore, the system has refined the business logic of the second layer of the hierarchical intent classifier. For the top-level category of "emergency response," the corresponding second-layer sub-classifier has been expanded into a classification network containing 80 subcategories, which precisely correspond to different combinations of disaster types and resource requirements. For example, the system can distinguish specific sub-intents such as "high-rise building fire - ladder truck requirement" and "chemical spill - decontamination team requirement," thereby achieving more accurate parsing of rescue instructions.

[0118] In the low-latency distribution phase of step S5, the system enhances the robustness of the communication link. The message queue module continuously monitors the heartbeat signal of the primary Ethernet link. Once a heartbeat signal loss is detected for more than 5 seconds, the system automatically determines that the primary network link is interrupted and immediately activates the backup LoRaWAN wireless communication link. At this time, the system extracts the core fields from the current order (including target coordinates latitude and longitude, required resource type code, and emergency code), generates a simplified order data packet, and sends it to the command center via the LoRaWAN link.

[0119] Meanwhile, complete order data will be persistently cached locally. Once the system detects that the Ethernet heartbeat has been restored, the cached complete order data will be automatically synchronized to the downstream system to ensure information integrity.

[0120] In the feedback loop, the system has added a key performance indicator—"task timeliness." This indicator is obtained by calculating the time interval between the successful placement of an order and the receipt of the "reinforcement arrived" status receipt from the execution system. The system presets a "golden 10 minutes" as a critical timeliness threshold for emergency response. If the calculated actual task completion time exceeds this threshold, the interaction is not only recorded as a delayed task, but the system also treats this event as a high-priority signal, triggering an emergency retraining process for the model to quickly optimize decision-making capabilities in time-sensitive scenarios.

[0121] Actual testing showed that this modified system maintained an intent recognition accuracy of 96.8% under 85 dB fire alarm noise and equipment free fall simulation tests, with an end-to-end latency of 148 milliseconds, fully meeting the stringent requirements of emergency command.

[0122] Example 3 This embodiment targets the rider terminal for chain fast food restaurants, using a customized Android phone. A typical command is "deliver the two Big Mac meals from table 3 to the south gate of Wanda Plaza." Due to limitations in the terminal's computing power, the system employs a model distillation and cloud-based collaborative strategy.

[0123] The voice front-end is simplified to a 4-channel microphone array, with the sampling rate reduced to 16 kHz. The feature extraction module retains only MFCC, and text encoding is replaced with a distilled version of TinyBERT. The core intent recognition model is compressed into a MobileNetV3-small architecture with <1M parameters. Environmental perception retains only the GNSS module, removing illumination and IMU. The task template engine only loads a subset of "food delivery".

[0124] The feature extraction processes in steps S1 and S2 are completed locally on the terminal. The terminal device first extracts Mel-frequency cepstral coefficient features of the speech, lightweight semantic features of the text, and simplified location features in parallel. Then, the terminal concatenates these three feature vectors to form a unified intermediate feature representation. This feature representation is uploaded to a regional edge cloud server deployed on the edge computing nodes of the 5G network via an encrypted channel established by Transport Layer Security (TLS) version 1.3.

[0125] After receiving the encrypted feature vector, the cloud server first decrypts and parses its format. The parsed features are then input into a larger-scale intent understanding model deployed in the cloud. This model is fine-tuned based on the BERT-large architecture to perform the fusion intent recognition in step S3 and the structured order generation in step S4. The cloud model, with its stronger computing power and larger parameter capacity, is specifically designed to handle complex instructions containing multiple items, complex requirements, or non-standard expressions.

[0126] To ensure the accuracy of complex instruction processing while minimizing overall latency, the system introduces an adaptive early exit mechanism. After completing local feature extraction, the terminal first uses a built-in lightweight intent recognition model to perform preliminary classification and confidence assessment of the current instruction. This lightweight model is a knowledge distillation version of the large cloud-based model, with significantly reduced parameters, specifically designed for quickly determining instruction complexity.

[0127] The system sets a confidence threshold, for example, 0.95. If the lightweight model's confidence in classifying the intent of the current instruction is higher than this threshold, the system determines that the instruction is a simple instruction. For such instructions, the system will no longer wait for cloud processing, but will directly call a simplified task template engine on the terminal to generate an order based on the extracted features and immediately enter the distribution process.

[0128] If the confidence level is below the threshold, it is determined to be a complex instruction, and the terminal will upload the features to the cloud for in-depth processing according to the aforementioned process. This mechanism realizes dynamic routing of processing paths.

[0129] Furthermore, the online optimization module was changed to a federated learning architecture: gradients are calculated locally on each terminal and aggregated daily to the central server to update the global model, protecting user privacy. After running for a period of time, the system's recognition rate for local snack names (such as "snail rice noodles" and "roujiamo") improved.

[0130] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0131] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A real-time voice order placement method based on multimodal artificial intelligence, characterized in that, include: In response to the user's voice order command, the system synchronously collects and time-aligns the user's voice, historical interaction text, and environmental awareness data to form a multimodal input. Parallel feature extraction is performed on the multimodal input, and the alignment of speech, text and environmental features in the semantic space is achieved through a cross-modal attention mechanism to generate a unified aligned feature representation. The aligned feature representation is input into a gated fusion network. The gated fusion network dynamically generates fusion weights based on the input quality of each modality, performs weighted fusion on the aligned features, and inputs the weighted fused features into a hierarchical intent classifier to output structured task intent labels. The corresponding structured task template is invoked based on the task intent tag, and key entity information is extracted based on the aligned feature representation to fill the task template, thereby generating a structured task order. The task orders are distributed with low latency through a high-priority in-memory message queue, and order execution feedback is collected synchronously. An online optimization loop is built based on the feedback to continuously optimize the performance of the gating fusion network and the hierarchical intent classifier.

2. The real-time voice order placement method based on multimodal artificial intelligence according to claim 1, characterized in that, Synchronously collect and time-align user voice, historical interaction text, and environment-aware data, including: The raw audio stream is acquired using a multi-channel microphone array at a preset sampling rate and divided into audio frames of fixed duration. Synchronously query the local interaction database to obtain historical interaction text records for a preset number of rounds, forming a text sequence that represents the current dialogue context; Environmental perception data is read from integrated multiple sensors at a fixed frequency, and the environmental perception data includes at least light intensity, geographical location and motion status data; The audio frames, the text sequences, and the environmental awareness data are each stamped with a high-precision timestamp provided by a precision clock synchronization protocol, and data is exchanged through shared memory to complete time alignment.

3. The real-time voice order delivery method based on multimodal artificial intelligence according to claim 1 or 2, characterized in that, Parallel feature extraction from multimodal inputs includes: After pre-emphasis, windowing, and short-time Fourier transform of the audio frame, the Mel frequency cepstral coefficients and filter bank energy characteristics are calculated and spliced ​​to form an acoustic feature vector. The text sequence is input into a pre-trained and fine-tuned semantic encoding model, which outputs a text semantic vector. After normalizing and organizing the time-series signals reported by environmental sensors, the signals are input into a one-dimensional convolutional neural network for feature extraction, and the output is an environmental context feature vector.

4. The real-time voice order placement method based on multimodal artificial intelligence according to claim 3, characterized in that, Semantic space alignment is achieved through a cross-modal attention mechanism, specifically as follows: Using the acoustic feature vector as the query and the text semantic vector as the key and value, calculate the cross-modal attention weights; The cross-modal attention weights are used to perform a weighted summation of the environmental context feature vector sequence, guiding environmental information to focus on the key semantics of the text description; The attention-weighted acoustic features, textual semantic features, and environmental features are concatenated along the feature dimension and then processed by layer normalization to generate the unified aligned feature representation.

5. The real-time voice order placement method based on multimodal artificial intelligence according to claim 4, characterized in that, The gated fusion network dynamically generates fusion weights based on the input quality of each modality, including: Collect feature indicators that reflect the input quality of each modality, including at least the real-time signal-to-noise ratio of speech, the length of the current text sequence, and the variance of environmental sensor data; The unified aligned feature representation and the feature index are input together into a modality confidence evaluation subnetwork. This subnetwork dynamically outputs three weight coefficients corresponding to speech, text and environment modalities. The weight coefficients satisfy normalization constraints and are within a preset empirical value range.

6. The real-time voice order placement method based on multimodal artificial intelligence according to claim 1, characterized in that, The hierarchical intent classifier uses a two-layer structure: The first layer is the Softmax classification layer, which is used to categorize user intents into preset top-level business categories; The second layer is a refinement classifier corresponding to the top-level business category, which is used to further identify specific sub-intents under the top-level business category and finally output a one-hot encoded vector corresponding to the structured task intent label.

7. The real-time voice order placement method based on multimodal artificial intelligence according to claim 1, characterized in that, The task template is populated by extracting key entity information based on aligned feature representations, including: The text semantic vector is input into a named entity recognition model specifically for order generation to identify and extract entity values ​​of location, item, and quantity from the text context; Real-time keyword detection is performed on the acoustic feature sequence, and standardized item or attribute identifiers are obtained by matching a predefined keyword pronunciation feature library. According to the predefined field-entity type mapping rules, the extracted entity values ​​and the standardized identifiers are filled into the corresponding fields of the task template, and the unfilled fields are filled using historical default values.

8. The real-time voice order placement method based on multimodal artificial intelligence according to claim 1, characterized in that, Low-latency distribution via a high-priority in-memory message queue includes: The structured task order is serialized into a compact binary byte stream; The binary byte stream is submitted to a lock-free high-priority message queue implemented based on a shared memory circular buffer; The message queue server thread sends the binary byte stream to the downstream task execution system in real time through a maintained TCP long connection using zero-copy technology.

9. The real-time voice order placement method based on multimodal artificial intelligence according to claim 1, characterized in that, Constructing an online optimization closed loop includes: Order execution feedback is collected from three dimensions: explicit user confirmation, downstream execution system feedback, and implicit signals automatically identified by the system. The feedback, along with the corresponding order data and multimodal feature vectors, is encapsulated into a feedback record and stored. Based on the feedback records periodically, a reinforcement learning algorithm is used to incrementally fine-tune the parameters of the gating fusion network and the hierarchical intent classifier, using task success rate and user confirmation rate as reward signals.

10. A real-time voice order placement system based on multimodal artificial intelligence, used to implement the real-time voice order placement method based on multimodal artificial intelligence as described in any one of claims 1-9, characterized in that, include: The multimodal data acquisition and preprocessing module is used to respond to voice order instructions, simultaneously acquire user voice, historical interactive text and environmental awareness data, and perform time alignment. The cross-modal feature extraction and alignment module is used to extract features from speech, text and environment in parallel, and to achieve semantic-level feature alignment through a cross-modal attention mechanism to generate a unified aligned feature representation. The multimodal fusion intent recognition module includes a gated fusion network and a hierarchical intent classifier. The gated fusion network is used to dynamically weight and fuse the aligned features based on the input quality. The hierarchical intent classifier is used to classify the weighted fusion features and output a structured task intent label. The structured task order generation module is used to call the task template according to the task intent tag, extract entity information from the aligned feature representation to fill the template, and generate a structured task order that passes the business logic verification. The low-latency distribution and online optimization module is used to distribute the task orders through a high-priority message queue, collect execution feedback, and drive the online optimization of the fusion intent recognition module based on the feedback data.

Images