An ESP32-S3 cold and hot dual-channel low-latency multi-modal AI toy control method, system, device and equipment carrying a streaming large model

Abstract

The application discloses an ESP32-S3 cold-heat dual-channel low-latency multi-modal AI toy control method, system, device and equipment loaded with a streaming large model, is applied to the technical field of embedded intelligent devices, human-computer interaction and large model application, and adopts an end-cloud cooperative cold-heat dual-channel architecture. Voice wake-up, VAD framing and event-triggered camera frame extraction are completed locally on the terminal, a lightweight binary event package is packaged, encrypted and uploaded, and low power consumption is realized through time-sharing scheduling and dynamic voltage regulation. The cloud end assembles the minimum context, adopts a streaming output and a first segment priority mechanism to stabilize the first character response latency, non-instantaneous tasks are divided into cold path asynchronous scheduling, and absolute preemption of the hot path is supported. The scheme fuses multi-source data hierarchical linkage multi-modal peripherals, bypasses execution of a white list tool without blocking a main link, simultaneously realizes human-device style separation management and full-dimension white list safety management and control, and gives consideration to accompanying experience and children scene compliance.

Description

Technical Field

[0001] This invention relates to the fields of embedded intelligent devices, human-computer interaction and large model application technology, and in particular to a control method, system, device and equipment for an ESP32-S3 dual-path low-latency multimodal AI toy equipped with a streaming large model. Background Technology

[0002] Existing large-scale model dialogue systems are mostly geared towards general intelligent assistants, desktop agents, and multi-turn task-oriented chatbots. The mainstream implementation logic is as follows: after a user initiates a request, the system sequentially performs a series of operations, including conversation orchestration, tool decision-making, historical content compression, long-term memory read / write, permission verification, and multi-agent scheduling. The system uses redundant prompts, integrating tool descriptions, role rules, massive historical dialogues, memory files, and runtime metadata. When external capabilities need to be invoked, a chain execution model is commonly adopted: model generation tool calls instructions – tool execution – model secondary integration output. Simultaneously, personalized personas, long-term memory, conversation logs, and data analysis are all embedded in the main interaction chain. While this type of solution is suitable for complex task scenarios, it shows significant limitations when directly applied to products such as AI companion toys and children's interactive toys.

[0003] The existing technology has at least the following technical defects: High and fluctuating interaction latency: The general agent architecture with multiple layers of logic processing will significantly increase the response time of the first character. The latency in pure chat scenarios will also be amplified, with some scenarios having a latency of more than 10 seconds, which seriously affects the instant companionship experience.

[0004] Fragmented multimodal interaction: Existing solutions only focus on text / voice dialogue and do not integrate peripherals such as cameras, screens, vibration motors, and buttons into a unified emotional expression system. As a result, toys lack anthropomorphic and warm interactive performance.

[0005] Tool calls block the main chain: Conventional tools such as weather, music, and volume control use multi-round chained call logic, which further aggravates the latency problem. However, most toy scenarios are just simple chat and do not require heavy tool scheduling processes.

[0006] Personalization management is unreasonable: Toy characters are fixed attributes of the product. The existing solution reads and writes long-term memory and style data in every round of dialogue, which increases the overhead of the main link; if user preferences are not recorded at all, the continuity of interaction will be lost.

[0007] High security risks: The system grants permissions such as file read / write, global network access, and system command execution, which do not comply with safety standards for children's toys and household consumer products, and the permission boundaries are too broad. Summary of the Invention

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A low-latency, multimodal AI toy control method based on the ESP32-S3 dual-path (hot and cold) streaming large-scale model includes: acquiring multimodal signals based on the ESP32-S3, performing wake-up detection, VAD audio framing, and fixed-point energy calculation locally; stopping frame extraction by default, only triggering single frame extraction during events, extracting frames at low frequency with second-level intervals during the interaction period, uploading after ROI cropping and compression, and then powering off and going into sleep mode after 60 seconds of inactivity; reducing power consumption through time-sharing task scheduling, peripheral power-off, and dynamic voltage adjustment; and encapsulating data into four-segment binary event packets, with a single packet payload. No larger than 2KB, uploaded with double encryption and CRC integrity verification; a minimum context is constructed in the cloud by assembling a fixed persona, short session history, in-session style, and device status, with a global hard cap of 900 tokens and priority pruning rules. Short history is retained for a maximum of 7 rounds and has a 120s time-series expiration mechanism; the large model streaming interface is called, and the first fragment is immediately sent after both length and semantics are determined, with a 400ms fallback timeout, and subsequent fragments are transmitted at adaptive intervals to stabilize the first character response latency; non-style merging and log storage are included. Real-time tasks are categorized as cold paths and managed using a FIFO main queue, with tasks of the same entry time prioritized. Scheduling only occurs when CPU, memory, and network loads simultaneously reach idle thresholds. Hot paths trigger absolute preemption, allowing interruptible tasks to be immediately suspended and breakpoints saved, while non-interruptible tasks have a maximum wait timeout of 200ms; if the timeout occurs, the task is terminated, rolled back, and marked for retransmission. Emotional confidence is calculated by weighting text semantics and device behavior data, and screen, vibration, and TTS parameters are linked according to a three-level threshold, with audio timestamps used to unify command alignment. Whitelisted tools are routed independently, with only the highest priority tool matched per round. Local control tools are executed directly, while cloud query tools have dynamic timeouts. Timeout loading of fallback messages does not block the main conversation. Fixed personas are fixed and cannot be modified. User preferences within a session are temporarily cached as style increments and take effect immediately. After the session ends, users are grouped by dimension, and conflicts are merged using both priority and timestamp rules. The settings are asynchronously persisted and retained for traceability and version rollback support. The principle of least privilege is followed throughout, with devices, networks, and tools all included in the whitelist. Hot updates only distribute configurations and do not load executable code.

[0009] A control system for ESP32-S3 dual-path low-latency multimodal AI toys equipped with a streaming large model is disclosed. The system is used to execute executable instructions to perform the aforementioned control method for ESP32-S3 dual-path low-latency multimodal AI toys equipped with a streaming large model.

[0010] A control device for an ESP32-S3 dual-path low-latency multimodal AI toy equipped with a streaming large model is provided. The device is used to execute executable instructions to perform the aforementioned control method for the ESP32-S3 dual-path low-latency multimodal AI toy equipped with a streaming large model.

[0011] Its beneficial effects are as follows: The terminal side locally completes voice wake-up, VAD frame segmentation, and fixed-point energy calculation. It employs event-triggered low-frequency camera frame extraction and ROI cropping and compression, encapsulating the data into a four-segment lightweight binary event package for encrypted upload. Low-power operation is achieved through time-sharing scheduling, peripheral power-off, and dynamic voltage adjustment. The cloud assembles a minimized context consisting of a fixed persona, short conversation history, in-conversation style, and device status. A global 900-Token hard cap and priority cropping rules are set, relying on a large-model streaming interface and a dual-judgment mechanism for the first segment to stabilize the first-character response latency. Non-real-time tasks such as style merging and log storage are placed in the cold path asynchronous scheduling, while absolute preemption is executed when hot path interactions are triggered. The solution is equipped with multimodal emotion linkage, whitelist tool bypass, persona / style separation, and a comprehensive security control mechanism, adapting to the needs of child companionship scenarios.

[0012] First, the decoupling of hot and cold paths and the minimized context design significantly reduce the latency of the first character response, solving the problems of high latency and large fluctuations in general large model architectures and ensuring an instant companionship experience. Second, lightweight preprocessing and low-power scheduling on the device side adapt to the hardware characteristics of the ESP32-S3 with limited computing power and memory, reducing the standby and running power consumption of the device. Third, multimodal emotion hierarchical linkage connects voice, screen, and vibration to achieve collaborative expression, breaking the limitations of isolated operation of peripherals and enhancing the warmth of humanized companionship. Fourth, the whitelist tool is executed in a bypass manner without blocking the main dialogue link, taking into account both functional expansion and low latency requirements, and adapting to the use scenario of toy-based casual conversation. Fifth, the hierarchical management of character style and the full-dimensional whitelist control take into account both role consistency and user personalization, while also meeting the safety and compliance requirements of children's toys. Attached Figure Description

[0013] Figure 1 A flowchart of a low-latency multimodal AI toy control method with ESP32-S3 dual-path cold and hot circuits and equipped with a large streaming model provided in this embodiment of the invention; Figure 2 A flowchart of cold and hot dual-path task scheduling and resource preemption provided in this embodiment of the invention; Figure 3 This is a schematic diagram of a module of an ESP32-S3 dual-path low-latency multimodal AI toy control system equipped with a large-scale streaming model, provided for an embodiment of the present invention. Detailed Implementation

[0014] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. Figure 1 This application describes an exemplary embodiment of an ESP32-S3 dual-path low-latency multimodal AI toy control method with a streaming large model.

[0015] In this application embodiment, a method for controlling ESP32-S3 dual-path low-latency multimodal AI toys equipped with a streaming large model is described, such as... Figure 1 As shown: The S101, based on the ESP32-S3, acquires multimodal signals and performs wake-up detection, VAD audio framing, and fixed-point energy calculation locally. The camera stops frame extraction by default, and only triggers a single frame extraction during events. During the interaction period, it extracts frames at low frequency with second-level intervals, uploads them after ROI cropping and compression, and then puts the camera into sleep mode after 60 seconds of inactivity. Power consumption is reduced through time-sharing task scheduling, peripheral power-off, and dynamic voltage adjustment. Data is encapsulated into four-segment binary event packets, with a single packet payload not exceeding 2KB, and is uploaded with double encryption and CRC integrity verification.

[0016] In one implementation, based on the ESP32-S3 hardware platform, local preprocessing, low-power management, and encrypted uploading of multimodal signals such as voice, image, gesture, and button input are completed. First, audio signal acquisition and fixed-point voice preprocessing are performed. The input is raw audio data acquired by an I2S interface microphone, with a basic sampling configuration of 16 kHz sampling rate and single-channel 16-bit PCM format. The core processing logic includes two parts: voice activity detection and frame segmentation, and fixed-point energy calculation. The frame segmentation rule sets the duration of a single audio frame to 32 milliseconds, corresponding to 512 sampling points, with adjacent frames overlapping by 16 milliseconds of sampling data. All energy calculations are uniformly converted to int16 fixed-point integer operations. In the processing flow, audio frames are continuously output only after a valid human voice signal is detected; silent segments are directly filtered and discarded. When the device is idle, continuous frame segmentation stops, retaining only low-power voice monitoring and wake-word detection capabilities. When a wake-word, button event, or voice energy exceeds a preset threshold, the complete frame segmentation process is resumed.

[0017] For example, when the toy is stationary, only a low-power wake-up detection program runs. The audio data collected by the microphone is used to perform a fixed-point energy calculation every 32 milliseconds. If the energy value is lower than a preset threshold, the environment is considered silent, and subsequent framing and uploading operations are not performed. When a wake word or voice energy exceeds the threshold, the complete framing process is immediately initiated, continuously generating audio frames with a 16-millisecond frame shift. Each frame is accompanied by a fixed-point energy value and a VAD marker, providing lightweight data for subsequent uploading and cloud processing.

[0018] This system performs event-triggered frame extraction and ROI cropping and compression. The input is the raw image frames from a low-resolution camera. The native hardware output frame rate is 10 frames per second. Only key frames are extracted and reported locally; continuous high-frame-rate transmission is not performed. The core processing logic includes three categories: event-driven frame extraction, adjacent frame difference comparison, and intelligent ROI cropping. By default, the camera stops continuous frame extraction. Five events—wake-up, button press, device pickup, face appearance, and scene change—can trigger single frame extraction. During continuous interactive sessions, low-frequency frame extraction is performed at intervals of 3 to 15 seconds, with a preferred interval of 5 seconds. Adjacent frame difference comparison determines the magnitude of image change by calculating the difference in the average grayscale values ​​of two frames. If the difference is less than a preset threshold, the current frame is discarded, and uploading and subsequent image-to-text processing are not performed. ROI cropping prioritizes locating the face region and cropping the face and its surrounding preset area. If no face is detected, the center area of ​​the image is cropped. After cropping, thumbnail compression is performed to reduce data size. When the device is idle for 60 seconds, all image frame extraction operations stop, and the camera hardware directly enters a power-off sleep state.

[0019] For example, when a user wakes up the toy by pressing a button, the system triggers a single frame capture from the camera. Once a face is detected in the image, the system automatically crops the face and its surrounding preset area, compresses it into a low-resolution thumbnail, and prepares it for upload. During subsequent continuous interaction, a frame is captured every 5 seconds. If the difference in the average grayscale value between two consecutive frames is lower than a preset threshold, it is determined that the scene has not changed, and the current frame is discarded without being uploaded. When the user stops interacting and the device remains inactive for 60 seconds, the camera hardware cuts off power, completely stopping image acquisition.

[0020] The system optimizes time-sharing scheduling for computing power and power consumption. The core scheduling logic is an RTOS-based priority task scheduling mechanism, combined with dynamic voltage regulation and peripheral management to achieve low-power operation. At the computing power scheduling level, audio acquisition, voice activity detection, and wake-word detection are classified as high-priority tasks, receiving priority allocation of CPU resources; image frame extraction, attitude calculation, and state planning are classified as low-priority tasks, interspersed in idle time slots to avoid preempting audio link resources. At the power consumption management level, five strategies are implemented: time-sharing power-off of peripherals, dynamic clock voltage regulation, intermittent wireless sleep, batch merging and reporting, and hierarchical sampling sleep. During idle periods, the MCU main frequency drops to 80 MHz, instantly increasing to 240 MHz upon interactive wake-up, and immediately decreasing the frequency after preprocessing; after event packet upload, Wi-Fi enters sleep mode, shortening the duration of high-power RF; multiple sets of attitude and button events within a short interval are merged into a single packet for upload, reducing the number of wireless wake-ups; after a 60-second pause, audio frame segmentation and image frame extraction cease, retaining only low-power button interrupt monitoring.

[0021] For example, when the toy is idle, the MCU's main frequency is stable at 80 MHz, and the camera and speaker amplifier are powered off, retaining only low-power button interrupts and voice monitoring functions. When the user wakes the device, the main frequency instantly increases to 240 MHz, prioritizing audio framing and wake-up detection tasks. During lulls in audio processing, device posture calculations and status statistics are interspersed. If two shaking actions are detected consecutively within a short period, the system merges and encapsulates the two events into a single data packet for uploading, avoiding two triggers of the wireless RF wake-up. After a single data upload is completed, the Wi-Fi module immediately enters sleep mode, awaiting the next data transmission request.

[0022] The lightweight event packet encapsulation and encrypted transmission utilize a three-layer mechanism: binary structured encoding, dual-encryption transmission, and integrity verification. The event packet employs a compact binary encoding format, discarding redundant characters typical of JSON, and is structured into four segments. The first segment is a fixed 16-byte header, storing the magic number, unique device identifier, session number, timestamp, and total event length. The second segment is a 1-byte event type identifier, distinguishing between audio, image, button gesture, and device status events. The third segment is the payload, storing the specific data for different event types. The fourth segment is a 2-byte CRC checksum, used to detect random bit errors during transmission. The maximum payload per packet is 2KB; exceeding this limit results in fragmented packet packaging. The encryption mechanism employs dual protection at the link layer and application layer. Wi-Fi and Bluetooth communication establishes an encrypted tunnel throughout using TLS 1.2 or TLS 1.3 protocols. The application layer payload uses AES-128-GCM for encryption and integrity protection. The device key is stored in a secure hardware eFuse storage area and is never exposed in plaintext. Anti-tampering and anti-forgery are achieved through TLS links, device identity authentication, and AES-GCM integrity verification.

[0023] For example, after a user performs a single key click, the system generates a corresponding key gesture event. The packet header contains the current device ID, session ID, and trigger timestamp; the event type is marked as a gesture event; the payload contains the action tag corresponding to the click; and a CRC checksum is appended to the end. The entire packet length is much less than 2KB. The data packet is first encrypted at the application layer using AES-128-GCM, and then uploaded to the cloud via a TLS encrypted tunnel. Upon receiving the data, the cloud first verifies the CRC code to confirm there are no transmission errors, then decrypts the link layer and application layer data sequentially, parses the specific key action, and then proceeds to the subsequent interaction process.

[0024] S102 uses cloud-based assembly of fixed personas, short conversation history, conversation style, and device status to form the minimum context. It sets a global hard cap of 900 tokens and priority pruning rules. Short history is retained for a maximum of 7 rounds and has a 120s time-series expiration mechanism. It calls the large model streaming interface, and the first fragment is sent immediately after the length and semantics are both determined, with a 400ms fallback timeout. Subsequent fragments are transmitted at adaptive intervals to stabilize the first character response latency.

[0025] In one implementation, considering the requirements of low-latency casual chat interaction scenarios with toys, a dual constraint of independent token caps for modules and dual expiration of short history is introduced. Threshold pre-configuration is implemented for four types of context components: fixed persona, short conversation history, in-conversation style, and structured device status. Adaptive rules for dynamically adding and deleting modules based on interaction scenarios are provided. For tool call scenarios, a simplified tool description of ≤80 tokens is temporarily added and automatically removed after execution. Specifically for low-latency casual chat interaction scenarios with toys, independent token caps are set for each of the four context components: fixed persona (280 tokens), short conversation history (total 420 tokens with a single round of dialogue not exceeding 60 tokens), in-conversation style (120 tokens), and structured device status (60 tokens). A dual cleanup rule for short history is introduced: time-series expiration cleanup is performed when a single interaction record exceeds 120 seconds from the current time; full session expiration cleanup is performed when the device disconnects a long connection or triggers sleep mode; satisfying either condition results in the cleanup of the corresponding history. The system dynamically adds or deletes context modules based on the current interaction scenario. In pure casual conversation scenarios, no tool description text is attached. In tool call scenarios, a minimal tool description of no more than 80 tokens is temporarily added. After the tool is executed, the description is automatically removed in the next round of conversation. In device hardware interaction scenarios, the short history of the session is compressed to a maximum of 3 rounds. At the same time, the structured device status field is expanded, and there is no need to add tool text.

[0026] For example, in purely casual conversation scenarios, the system retains only four modules: fixed persona, short history of the last 7 rounds of conversation, conversation style blocks, and core device status, maximizing the reduction of invalid token overhead. When a user inquires about the intent to trigger a weather-triggered tool, the system temporarily appends a 72-Token weather tool description to the end of the context, including the tool's functions and return format rules. After the tool is executed in this round of conversation, the tool description is automatically removed in the next round, avoiding long-term occupation of context space. When a user shakes the device to trigger hardware interaction, the system automatically compresses the short history of the conversation to the most recent 3 rounds, while fully reporting device fields such as grip posture and vibration status, without adding extra tool descriptions. When the device is idle for more than 60 seconds and triggers sleep mode, all short histories are directly cleared from the real-time cache, and the original complete conversation record is transferred to background asynchronous storage without losing any interaction data.

[0027] Based on the optimization goals of a global hard cap of 900 tokens and no loss of core interaction information, the context content is pruned using a four-level priority decreasing pruning method. After pruning, the total length is re-verified, and a minimal context sequence with total quantity control, hierarchical pruning, and scenario adaptation is constructed. Expired short history data is transferred to a cold path and asynchronously written to log storage, ensuring no loss of complete interaction data. Using a global hard cap of 900 tokens as the overall constraint and no loss of core interaction information as the optimization goal, the context content is pruned according to a four-level priority decreasing rule. The priority, from lowest to highest, is: short history of old sessions and secondary device status, synonymous modifiers in long-sentence style descriptions, non-critical supplementary statements in the latest session, fixed personas, and core information of the current round. The lowest priority content is removed first. After each round of pruning, the total token length is re-verified until the threshold requirement is met. The pruning process only removes decorative and historically redundant content, without affecting the current core interaction requirements and basic role settings, ultimately constructing a minimal context sequence with total quantity control, hierarchical pruning, and scenario adaptation. Expired short history data that has been cleaned up is not destroyed directly, but is transferred to a cold path and asynchronously written to the log storage, retaining complete interaction data for subsequent analysis.

[0028] For example, when a user's consecutive conversations result in a total context token count of 920, exceeding the global hard cap of 900 tokens, the system first performs a first-level pruning, removing the earliest old conversation record and secondary lighting status fields. After pruning, the total token count is 905, still exceeding the threshold. Next, a second-level pruning is performed, streamlining descriptive terms in the style description while retaining core keywords such as address and tone. After pruning, the total token count is 892, meeting the threshold requirement. Pruning then stops, and the final context sequence is generated. The removed old conversation records are not directly deleted but are synchronously transferred to a cold path and written to the backend log storage, which can be used later for user preference analysis and conversation replay.

[0029] First, the assembled minimum context is sent to the large model streaming inference interface. Then, the validity of the first segment is checked. The first packet delivery trigger mapping is completed based on the dual judgment of length threshold and semantic integrity. Utility class responses are adapted to dedicated judgment rules, with a 400ms fallback timeout forced delivery mechanism. The semantic priority segmentation rule of 8~32 tokens and the three-level adaptive interval adjustment logic are determined. The timing of text and audio segments is bound, and low-latency streaming output control information adapted to the ESP32-S3 embedded terminal is generated. The assembled minimum context is sent to the large model streaming inference interface, and the validity of the first segment is checked. The first packet delivery is triggered based on the dual judgment rules of length threshold and semantic integrity. The length threshold requires the first segment to be at least 8 tokens. The semantic integrity requires the segment to contain sentence-ending punctuation or form an independent subject-predicate semantic unit. Utility class responses that reach 8 tokens and are semantically independent can be delivered directly. A 400ms fallback timeout forced delivery mechanism is provided. After the timeout, the current content is delivered regardless of whether the semantics are complete. Subsequent generated content is fragmented according to a semantic priority rule of 8 to 32 tokens, prioritizing fragmentation at punctuation marks and prohibiting forced splitting within words. Continuous long text without punctuation reaching 32 tokens is forcibly fragmented. The system adaptively adjusts the fragmentation interval in three tiers based on network bandwidth and device load: 80ms interval when downlink bandwidth is greater than 50kb / s and device CPU load is below 40%; 120ms baseline interval when bandwidth is between 20 and 50kb / s and CPU load is between 40% and 70%; and 220ms interval when bandwidth is less than 20kb / s and CPU load is above 70%. When the large model inference ends and there is no remaining text, the interval is canceled and the final concluding fragment is immediately sent, while simultaneously binding the timing relationship between the text fragment and the corresponding audio fragment to ensure synchronized playback on the terminal.

[0030] For example, after a user initiates a casual conversation, the large model streams content. When the 10th token is generated and a period appears at the end, the system determines that both length and semantic integrity requirements are met, and immediately sends the first segment to the device, simultaneously initiating TTS synthesis and playback. If the model continues to generate long sentences and fails to form a complete semantic unit within 400 milliseconds, the system triggers a fallback timeout mechanism, directly sending the currently generated text as the first segment to ensure stable latency perceived by the user. During subsequent generation, if the device is detected to be in a weak network and high-load state, the segment sending interval is automatically extended from 120 milliseconds to 220 milliseconds to reduce high-frequency small packet interactions and lower the probability of packet loss. When the large model outputs the last segment, it sends the final segment directly without waiting for an interval. Each text segment is synchronously bound to a corresponding 200-millisecond audio segment to ensure accurate synchronization of voice and visual actions during playback on the device.

[0031] S103 classifies non-real-time tasks, including style merging and log storage, into cold paths and manages them using a FIFO main queue. Tasks with the same enqueue time are sorted by priority. Scheduling only occurs when the CPU, memory, and network loads all reach the idle threshold simultaneously. When a hot path is triggered, absolute preemption is executed. Interruptible tasks are immediately suspended and breakpoints are saved. Uninterruptible tasks are set to a maximum waiting timeout of 200ms. If the timeout occurs, the task is terminated, rolled back, and marked for retransmission.

[0032] In one implementation, considering the low-latency priority scheduling requirement of AI toy hot paths, a dual constraint of task priority grading and resource load threshold is introduced. This allows for queuing and lifecycle management of non-real-time tasks, including style merging, log storage, and device diagnostics. Only high-value tasks such as style merging at the end of a session and fault diagnosis are allowed to skip queues. For non-real-time tasks like style merging, log storage, and device diagnostics, a dual constraint of task priority grading and resource load estimation is introduced. All tasks undergo standardized marking and lifecycle configuration upon queuing. Each task is bound to a unique task ID, task type tag, estimated resource consumption, expiration timestamp, and priority level upon queuing. Priorities, from highest to lowest, are: user style persistence, session summary generation, log shard storage, usage statistics, and device diagnostic reports. Tasks with the same queuing time are sorted by priority. The system restricts queuing permissions to only two types of high-value tasks: style merging tasks immediately after a session ends and fault diagnosis tasks. Only one task is allowed to skip queues at a time; all other tasks are appended to the end of the queue.

[0033] For example, after a user completes a full conversation, the system generates a conversation style merging task. This task is marked as a high-priority queue-jumping type and directly inserted at the head of the cold task queue, waiting for scheduling first. Simultaneously, the generated conversation usage statistics task is marked as a normal low-priority type and appended to the tail of the queue, waiting for execution in the order of queuing during an idle window. Regular daily device diagnostic report tasks, after entering the queue, are by default placed after all conversation-related tasks and are only scheduled when there are no higher-priority tasks.

[0034] Based on the optimization goals of zero-blocking hot links and controllable background resources, cold tasks are tiered and filtered. A standardized cold task scheduling sequence is constructed, featuring FIFO main queue sorting, priority queueing for certain categories, and automatic destruction of expired or overdue tasks. Five types of destruction rules are provided: expiration, session expiration, resource overrun, deduplication, and preemption timeout. With the optimization goals of zero-blocking hot links and controllable background resources, cold tasks within the queue are tiered and filtered, and a standardized scheduling sequence based on a first-in-first-out main queue is constructed, along with five types of automatic destruction rules to complete the closed-loop lifecycle of tasks. The first category is expiration destruction. Tasks are configured with a 15-minute effective lifespan when enqueued, and are automatically destroyed if they are not scheduled after the timestamp expires. The second category is session expiration destruction. When the corresponding session is disconnected and the device is offline for more than 5 minutes, all cold tasks associated with that session are destroyed. The third category is resource overload destruction. When the total number of tasks in the queue reaches 60, the lowest priority statistical report tasks at the end of the queue are destroyed first. The fourth category is deduplication destruction. Duplicate logs and duplicate summary tasks generated by the same device and the same session are automatically merged, and redundant copies are destroyed directly. The fifth category is preemption timeout destruction. When a cold task is preempted by a hot path multiple times and the cumulative suspension time exceeds 10 minutes, it is determined to have no execution value and is destroyed.

[0035] For example, if the system is under continuous high interactive load and no complete idle window appears for 12 consecutive minutes, and the earliest enqueued device diagnostic report task reaches its 15-minute lifespan threshold, the system will automatically destroy it and release the corresponding memory resources. If a toy device is disconnected from the network for more than 5 minutes, all pending tasks for the corresponding session summary and style merging tasks will be removed from the queue to prevent invalid tasks from occupying queue space for a long time. When the same device triggers the same log reporting task three times in a short period of time, the system will automatically merge them into a single task, and the other two redundant copies will be destroyed directly.

[0036] First, the system verifies that the CPU, memory, and network loads are all within acceptable limits and there is no active hot interaction. Then, it initiates parallel scheduling of cold tasks. It relies on the absolute preemption mechanism of hot paths to complete resource conflict mapping, adapts to the handling logic of interruptible task suspension at breakpoints and non-interruptible tasks waiting for safe interruption points, determines the rules for 200ms timeout termination rollback and retransmission marking, and implements multi-layered parallel resource upper limit constraints. This generates implementation information for the coordinated scheduling of cold and hot paths adapted to the ESP32-S3 toy terminal. The system first performs idle state verification. Only when the CPU, memory, and network load thresholds and the supplementary condition of no active hot interaction are met does it initiate parallel scheduling of cold tasks. It also implements multi-layered parallel resource upper limit constraints and dynamic scaling strategies to prevent background tasks from crowding out hot path resources. The idle status criteria are as follows: the average CPU utilization rate of the sliding window in the past second does not exceed 30%, and the peak utilization rate of a single core does not exceed 45%; the remaining available memory of the system is not less than 35% of the total memory; the instantaneous uplink and downlink bandwidth utilization is less than 20% of the total capacity; and there are no running hot path interaction tasks, pending real-time voice and image input packets, or pending multimodal control commands. If any one of these criteria is not met, the system is judged to be busy, and the scheduling of new cold tasks is suspended.

[0037] During parallel scheduling, multi-level resource limits are set. A maximum of three cold computing tasks can run concurrently at any given time, with a single task's CPU usage capped at 25% and the total CPU usage of multiple tasks not exceeding 30%. Only two cold tasks are allowed to perform network transmissions simultaneously, with a total bandwidth usage capped at 20% of the total capacity. The total memory usage of all parallel cold tasks cannot exceed 25% of the remaining available memory. Only a single cold task is allowed to perform persistent write operations at any given time. When a concurrent cold task triggers the resource limit, new cold tasks are stopped from scheduling and pushed into the queue. If the load gradually increases, the system automatically reduces the number of parallel cold tasks, pausing them starting with low-priority reporting and statistics tasks. Long-queued cold tasks are given priority for parallel resource allocation in the next system idle window to prevent low-priority tasks from being starved for extended periods.

[0038] When any one of the following resources—CPU instantaneous usage, network bandwidth, or memory buffer—reaches the concurrency threshold, and there are both pending hot tasks and running cold tasks simultaneously, a path conflict is identified, and the hot path absolute preemption mechanism is executed. Interruptible cold tasks are immediately paused, the task breakpoint context is saved, and all resources are released. Non-interruptible cold tasks wait to reach a safe breakpoint, with a maximum wait time of 200 milliseconds. If the wait time expires, the cold task is terminated, a transaction rollback is performed, and it is marked as a task to be retransmitted. After the hot path task completes execution, the system reads the breakpoint data of the suspended cold tasks and resumes execution according to the original queuing order.

[0039] For example, when the system is idle, it might simultaneously run two cold tasks: log sharding upload and user preference extraction. If a user then initiates a dialogue by pressing a button to wake up a toy, the hot path interaction task is started. The system determines a resource conflict. Since user preference extraction is an interruptible in-memory computation task, it immediately saves the computation breakpoint and suspends the task, releasing the occupied CPU and memory resources. Log upload, in its uninterruptible write phase, waits for it to reach a safe breakpoint. After 190 milliseconds, the write operation completes, and the log upload task is suspended, with all resources allocated to the hot path interaction. After this dialogue ends, the system reads the breakpoint data from both cold tasks and resumes execution in their original priority order, without having to start the tasks from scratch. If subsequent rounds of high load cause a statistical report task to queue for a long time, this task will be prioritized for parallel resource allocation when the system has an idle window, preventing it from being unable to execute for an extended period.

[0040] S104 integrates text semantics and device behavior data to calculate emotion confidence, and links screen, vibration and TTS parameters according to three-level thresholds, and aligns instructions uniformly with audio timestamps; whitelisted tools are independently bypassed and routed, and only the highest priority tools are matched in a single round. Local control tools are directly issued and executed, while cloud query tools are set with dynamic timeouts. Timeout loading of fallback dialogue does not block the main dialogue.

[0041] In one implementation, combining the optimization requirements of multimodal anthropomorphic interaction of AI toys and low latency of the main link, a dual-branch weighted fusion of text (0.7) and behavior (0.3) and a tool whitelist classification control are introduced as dual constraints. This preprocesses and prioritizes six types of emotional features and tool request intentions, abandoning the general multi-turn chained call mode of agents. The input data consists of two parts: one is the complete dialogue text of the current user, from which basic emotional features are extracted by a lightweight cloud-based emotion classification model, outputting raw confidence scores for six basic emotions: happy, depressed, calm, curious, irritable, and neutral; the other part is the structured state variables reported by the device, which map corresponding emotion weights based on behaviors such as continuous shaking, repeated key presses, prolonged silence, and frequent follow-up questions. The fusion calculation is weighted according to a ratio of 0.7 for text and 0.3 for behavior, finally outputting a normalized emotion confidence score in the range of 0 to 1. The tool matches user request intents based on a predefined whitelist, marks local control and cloud query attributes according to function type, and simultaneously marks execution priority. Only the highest priority tool requests are retained in a single round of dialogue.

[0042] For example, if a user's dialogue expresses happiness while repeatedly shaking a toy, the text branch identifies an initial confidence score of 0.8 for happiness, while the behavior branch maps a happiness weight of 0.7 based on the continuous shaking. After proportional weighting and fusion, the final happiness confidence score is 0.77. Simultaneously, if the user's request does not match any tool whitelist keywords, the tool invocation process is not triggered, and all calculation results are synchronously sent to the subsequent hierarchical filtering stage. If the user mentions both increasing the volume and checking the weather, after matching both tool requests, the system retains the higher-priority local volume adjustment command and discards the lower-priority weather query request to avoid multiple tools consuming link resources in parallel.

[0043] Based on the goals of graded adaptation of emotional confidence and zero-blocking main dialogue for tool invocation, a graded screening of emotional linkage strategies and tool execution paths is implemented. A standardized scheduling sequence with three-level confidence thresholds and single-round highest priority matching is constructed, clarifying the priority of emotional conflicts and the execution rules for local / cloud tools. Emotional switching waits for the end of the current audio sentence to avoid parameter jumps within the same line. With the optimization goals of graded adaptation of emotional confidence and zero-blocking main dialogue for tool invocation, a graded screening of emotional linkage strategies and tool execution paths is implemented, and a standardized scheduling sequence is constructed. Three-level confidence thresholds are set on the emotional side. A final confidence score greater than or equal to 0.7 is judged as high-confidence emotion, and the corresponding full multimodal linkage solution is fully executed, including dedicated screen animation, vibration mode, and quantitative adjustment of voice parameters; a confidence score between 0.4 and 0.7 is judged as low-confidence emotion, and only simplified voice tone fine-tuning is performed, while the screen and vibration remain in a basic calm mode; a confidence score below 0.4 is judged as no obvious emotion, and a fixed neutral basic multimodal output is used, without loading emotion-specific animations and vibrations. If the scores of multiple emotion fusions are close and the difference does not exceed 0.08, they are executed according to a fixed priority order, from highest to lowest: depressed, happy, irritable, curious, and neutral. Only the linkage strategy of the highest priority emotion is triggered. The emotion switching operation waits until the current audio sentence ends, and the entire set of parameters is switched only when the sentence pause is no less than 300 milliseconds, to avoid frequent jumps in animation and vibration parameters within the same line of dialogue. The tool clearly defines two types of execution rules: local control and cloud query. All tool requests run along independent paths and are not embedded in the main inference chain.

[0044] For example, if a user expresses a depressed mood and the fused confidence score is 0.75, reaching the high confidence threshold, the system will match a full set of depressed mood-related strategies, including a soft, half-closed screen with gentle eye movements, long-cycle low-intensity vibrations, a slower speech rate, and a lowered tone. If the difference between the scores for depressed and happy moods is 0.06, the system will execute only the depressed mood-related strategy according to priority rules, ignoring the lower-priority happy mood. If the user's mood changes from depressed to happy during the conversation, the system will wait for a pause of more than 300 milliseconds in the current audio sentence before switching to the full set of multimodal parameters to ensure no parameter jumps during dialogue playback.

[0045] First, using the TTS audio timestamp as a global benchmark, the screen animation and vibration commands are aligned through three layers of timing: preloading, synchronous execution, and termination. Then, an independent bypass route for the tool intent is initiated. Multimodal parameter quantization mapping is completed based on emotion grading rules. This adapts to three levels of dynamic timeout handling logic: local tool direct execution and cloud query. Timeout fallback dialogue and anomaly degradation rules are determined. Tool results are only used as supplementary content in the dialogue and are not embedded in the main inference chain. This generates emotion-tool collaborative scheduling implementation information adapted to the ESP32-S3 toy terminal. Timing alignment is divided into three stages: 150 milliseconds before audio playback, a pre-loading command is issued to complete the initial eye image on the screen and vibration pre-start buffer; at the start of the actual audio playback, TTS output, animation playback, and vibration cycle loop are started synchronously; 200 milliseconds after the audio ends, vibration stops synchronously, and the screen switches back to a neutral static eye image. The alignment error between the screen animation frame and the audio playback timestamp is controlled within 15 milliseconds. If the error exceeds this range, the lagging animation frame is automatically discarded. The deviation between the vibration waveform start point and the audio timestamp of the first word of the dialogue does not exceed 50 milliseconds, ensuring the synchronization of multimodal output.

[0046] During the tool execution phase, local control commands are directly sent to the ESP32-S3 terminal for execution. Preset templates are used to generate independent short sentences that are appended to the beginning of the response, eliminating the need to wait for model inference to complete. Local control tools do not have a retry mechanism. Cloud-based query tools have dynamic timeout thresholds. The gateway collects the average round-trip latency and timeout failure rate of the tool's interface over the past 5 minutes in real time, adjusting the actual timeout value in three tiers. When the failure rate is below 5%, the baseline timeout is maintained: 800 milliseconds for lightweight interfaces and 1200 milliseconds for lifestyle service interfaces. When the failure rate is between 5% and 15%, the timeout value is increased by 30% to provide a buffer. When the failure rate is above 15%, the timeout value is reduced to 0.4 times the baseline value to quickly trigger a fallback response. The dynamically adjusted timeout value has upper and lower limits, with a minimum of 300 milliseconds and a maximum of 3000 milliseconds to avoid extreme thresholds. The cloud-based query tool supports only one retrieval, with a retry interval of 200 milliseconds. If the retry fails, a preset fallback and reassurance message is loaded directly. When the tool returns a normal result, structured data is populated into the output template as preliminary information, followed by casual conversation statements generated by the model. The two parts flow naturally without disrupting the semantics. All tool exceptions are asynchronously archived in a cold path, without consuming main link computing power.

[0047] For example, the system generates a response indicating a happy mood and simultaneously calculates the corresponding multimodal parameters. 150 milliseconds before TTS audio playback, the cloud sends an initial smiling eye image and vibration pre-start command to the device; when the audio playback begins, voice broadcasting, animation playback, and a high-frequency short vibration cycle are simultaneously initiated; when the audio playback reaches the last 200 milliseconds, the vibration stops synchronously, and the system switches back to a neutral, static eye image. Throughout playback, the alignment error between the screen frame and the audio remains stable at 12 milliseconds, and the deviation between the vibration start point and the first word of the audio is 30 milliseconds, both meeting the accuracy requirements.

[0048] If a user triggers the weather query tool, the system independently initiates a cloud request. Given a stable network and a timeout failure rate below 5%, a baseline timeout of 1200 milliseconds is maintained. If the request times out without a response, the system triggers a retry at 200-millisecond intervals. If the retry also fails, a pre-set fallback reassurance message is loaded and simultaneously sent to the device for playback, without waiting for the tool's result. Subsequent data returned by the tool is naturally supplemented in the next round of dialogue, without interrupting the streaming output of the current main dialogue. When a normal result is returned, the city temperature and weather conditions are displayed at the beginning of the template, followed by a travel suggestion sentence generated by the model. These two sections are then seamlessly combined into a complete response.

[0049] S105 features fixed and unmodifiable user personas. User preferences within a session are temporarily cached as style increments and take effect immediately. After the session ends, users are grouped by dimension, and conflicts are merged using both priority and timestamp rules. The system is asynchronously persisted and retained for traceability, supporting version rollback. The principle of least privilege is followed throughout the process, with devices, networks, and tools all included in the whitelist. Hot updates only distribute configurations and do not load executable code.

[0050] One implementation combines the need for unified AI toy character positioning with user personalization and the need for safety and compliance in children's scenarios. It introduces a dual constraint of layered isolation of character styles and minimum permissions across all dimensions. This involves pre-configuring and lifecycle marking of fixed character content, incremental conversation styles, and permission boundary rules. It also clarifies the core constraint that hot updates only distribute whitelist configurations, parameters, and templates, and do not load executable code.

[0051] The character layer is predefined by the product, including role positioning, basic tone, worldview, and taboo content. It is fixed in the system and cannot be modified. Each round of dialogue only loads a fixed module and is not rewritten in real time. The style increment layer records the user's personalized preferences in the conversation, divided into four dimensions: address, tone, sentence structure, and taboos. Each increment is bound to a unique number, conversation number, generation timestamp, cache expiration time, priority, and activation flag. The cache expiration time is set to 20 minutes after the end of the conversation, and it is automatically cleared from memory after the timeout. At the permission boundary level, devices, networks, and tools are all included in whitelist management. Devices are prohibited from executing system commands and reading and writing files across the entire domain. Network access is only allowed to connect to preset server addresses. Tool capabilities are strictly limited to a predefined range. Hot updates only support updating tool whitelist configurations, trigger keywords, timeout parameters, and output templates. Executable code is not dynamically loaded throughout the process, and all configuration files include version numbers and signature information.

[0052] For example, products are pre-set with a friendly, companionable cartoon character, including a fixed tone and dialogue taboos, which users cannot modify through dialogue. If a user requests to address the user as "Xiaoyu" and keep their speech brief, the system extracts this request as style increments in terms of both address and sentence structure, binds it to the current session number and timestamp, marks it as effective immediately, and stores it in the session memory cache. Regarding access control, devices can only connect to three designated cloud servers by default, and tools only have preset functions such as volume adjustment and weather queries enabled; they cannot access other network addresses or perform unauthorized operations. When the tool triggers keywords in the background, only the configuration parameter file is sent, not any executable program. The device verifies the version number and updates the rules without requiring a system restart.

[0053] To achieve the optimization goals of zero-blocking main link for style persistence and no unauthorized access control throughout the process, style increments and security rules undergo hierarchical verification and screening. A standardized style management sequence is constructed, grouped by four dimensions, with conflicts determined by priority and timestamps, and featuring source tracing, retention, and rollback capabilities. This is complemented by a three-level whitelist security control system for devices, networks, and tools, along with content filtering. Style version snapshots are retained for a minimum of 90 days, and historical version rollback is supported.

[0054] At the style processing level, multiple style increments within the same session are grouped and processed according to four dimensions. Increments within the same dimension are judged for conflict based on both priority and generation timestamp rules. Semantically mutually exclusive features are mainly increments with higher priority and more recent timestamps, and the overwritten old increments are stored in the source field for retention. Features that are partially overlapping and not mutually exclusive are directly merged and coexisted without triggering overwriting. When more than three mutually exclusive increments appear in the same dimension in a single session, all source information is retained, and only the last high-priority increment is used as the currently effective style. In terms of security management, in addition to the three-level whitelists for devices, networks, and tools, both model inputs and outputs undergo field-level content filtering to block illegal and sensitive content. After each style incremental merge is completed, a version snapshot is automatically generated, which is bound to the version number, merge time, and incremental traceability list. All snapshots are retained for a minimum of 90 days, and the merge records of the past 30 days are fully retained. Records from 30 to 90 days ago are automatically compressed and deleted with detailed traceability information. Records older than 90 days are deemed expired and are asynchronously cleaned up. Style records that have not been reused in any sessions in the past 30 days are compressed in advance, and those that have not been accessed for 120 consecutive days are directly deleted. During the retention period, rollback and retrospective operations are supported.

[0055] For example, in the same conversation, a user requests both a more lively tone and a gentle, quiet tone. These two increments, both belonging to the tone dimension, are semantically mutually exclusive. The system compares priorities and timestamps, determining that the gentle, quiet tone request has higher priority and is set as the currently effective style. The previous lively tone increment is stored in the source field for future reference and is not directly deleted. In terms of security, both user input and model output text undergo content filtering to block content unsuitable for children. After one week of use, six style version snapshots are generated in the cloud, all within the 90-day retention period. When a user issues a command to restore the previous speaking style, the system can select the corresponding version from the historical snapshots, overwrite the current conversation's style increment with the core style features, and complete the rollback operation. If a style record has not been accessed by any conversation for 40 consecutive days, the system automatically compresses its detailed source data, retaining only the core style features.

[0056] First, style increments are cached in session memory and applied immediately without being written to the persistent layer. After the session ends, they are transferred to a cold path for merged persistence. Conflict verification mapping is completed based on rules of mutual exclusion and overlapping fusion. This adapts to the handling logic of dual-media storage (edge ​​and cloud) and configuration signature distribution. Version snapshot retention and anomaly security degradation rules are determined. The key is stored in a hardware secure area. All configuration distributions are verified by digital signatures. This generates style security integrated management and control implementation information adapted to the ESP32-S3 toy terminal.

[0057] Style persistence employs a dual-media architecture (endpoint and cloud). The cloud primarily uses a distributed key-value database with read / write separation nodes. Hot-read nodes handle session loading and queries, while cold-write nodes handle batch persistence for cold paths, without consuming main interaction database resources. On the device side, a dedicated Flash partition serves as a fallback, allocating 256KB of space to store only incremental snapshots of currently active sessions, omitting full history. In case of network interruption, a local snapshot is written first, and then uploaded to the cloud after network recovery. Merging operations are performed only during cold paths when sessions are closed, devices are idle, and network load is low. After merging, a unified persistent style record is generated, along with an incremental traceability list, and a version snapshot is generated simultaneously. Version rollback only affects the current session's memory cache. For permanent rollback, a cold merge must be triggered upon session termination to generate a new persistent version.

[0058] At the security execution level, device keys are stored in the hardware eFuse secure zone and are not exposed to the application layer in plaintext. All whitelist configurations, style configurations, and output templates must be digitally verified; if verification fails, updates are rejected. When the device-side Flash partition usage reaches the 80% threshold, the oldest offline snapshot is automatically discarded. After a single session ends and cloud synchronization is successful, all local temporary incremental snapshots are cleared, retaining only the minimal snapshot.

[0059] For example, after a user ends a conversation, the system transfers the three style increments generated in that session to the cold path for processing. It groups these increments by dimension, merges conflicts, generates the latest persistent style record, and simultaneously generates a corresponding version snapshot, storing it in the cloud database. If the device's network suddenly drops, the system first writes the merged style snapshot to the local Flash partition as a fallback. Once the network is restored, it automatically re-uploads it to the cloud to prevent style data loss. When a user triggers a style rollback, the selected historical version only overwrites the current session's memory cache and does not modify the persistent data in the cloud. If the user confirms permanent use of the version, the system will re-execute the merge during the cold path phase after the current session ends, generating a new persistent version. When the backend issues new tool whitelist configurations, the configuration file includes a digital signature. The device first verifies the signature validity using the key in the hardware security zone. Only after successful verification is the local rule updated; otherwise, the configuration package is discarded to prevent unauthorized tampering.

[0060] like Figure 2 As shown, Figure 2 This is a flowchart of a cold and hot dual-path task scheduling and resource preemption process, which shows the full-process scheduling logic of the cold and hot dual-path architecture of this solution. It covers four core links: cold task queuing management, system load determination, hot path resource preemption, and parallel resource constraints. The core principle is to prioritize low latency hot paths, execute cold tasks asynchronously during idle periods, and ensure that hot tasks are absolutely preempted when there is a resource conflict.

[0061] New Cold Task Enqueueing and Lifecycle Management (Right-Side Link): When a new background cold task enters the scheduling queue, it is processed step-by-step according to priority and lifecycle rules. First, priority is determined: if it is a high-priority task that can be queued, it is directly inserted at the head of the queue; ordinary tasks are appended to the tail of the queue. Then, the destruction conditions are checked: if any one of the following conditions is met—expiration, session expiration, queue exceeding limit, task duplication, or preemption timeout—the task is directly destroyed to release resources; tasks that have not triggered destruction wait for the system to retrieve them for execution in an idle window. Before execution, the concurrent resource limit is checked: if the current concurrent cold tasks have reached the resource limit, new tasks are paused and low-priority tasks are suspended; if the limit has not been reached, multiple tasks are executed in parallel over time.

[0062] The system performs idle state load detection (intermediate link). It monitors the CPU, memory, and network loads in real time. A system is considered idle and allowed to schedule cold tasks only if all four conditions are met: CPU utilization ≤ 30%, remaining available memory ≥ 35%, bandwidth utilization ≤ 20%, and no currently active hot path tasks. If these conditions are met, cold tasks are scheduled normally; otherwise, no new cold tasks are added, and existing interruptible cold tasks are suspended to reserve resources for hot path operation.

[0063] Hot and cold task conflicts and preemptive execution (left-side link): When hot and cold tasks are triggered simultaneously and resource conflicts occur, the hot path's absolute preemptive mechanism is executed. First, a resource conflict determination is made: if there is no conflict, hot and cold tasks execute in parallel; if a conflict exists, it further determines whether the cold task is interruptible. If the cold task is interruptible: the cold task's execution breakpoint is immediately saved, the task is suspended and returned to the queue, and all resources are made available to the hot task; after the hot task completes execution and releases resources, the breakpoint is read to resume the cold task's execution. If the cold task is uninterruptible: the hot task waits for a maximum of 200ms until the cold task reaches a safe interrupt point; if the wait times out, the cold task is terminated directly, a transaction rollback is performed, and resources are released for the hot task to use.

[0064] like Figure 3 As shown, an ESP32-S3 dual-path low-latency multimodal AI toy control system with a large-scale streaming model includes: The edge-side multimodal preprocessing module 301 is used to acquire multimodal signals based on ESP32-S3, locally complete wake-up detection, VAD audio framing and fixed-point energy calculation, control camera event trigger frame extraction, low-frequency extraction during interaction period and ROI clipping and compression, and put the device into a 60-second power-off sleep mode. It reduces power consumption through time-sharing scheduling, peripheral power-off and dynamic voltage adjustment, and encapsulates four-segment binary event packets, which are then uploaded to the cloud after double encryption and CRC integrity verification. The minimum context streaming dialogue module 302 is used to assemble a fixed persona, short conversation history, conversation style, and device status in the cloud to form a minimum context. It sets a global hard limit of 900 tokens and priority pruning rules, matches a 7-round short history limit and a 120s time sequence expiration mechanism, calls the large model streaming interface, and sends the first segment after dual judgment of length and semantics. It is equipped with a 400ms fallback timeout and adaptive segmented transmission to stabilize the first character response latency. The cold and hot path collaborative scheduling module 303 is used to classify non-real-time tasks such as style merging and log storage into the cold path. It adopts FIFO main queue to sort and manage tasks according to priority. Scheduling is only performed when the CPU, memory and network loads simultaneously reach the idle threshold. When the hot path is triggered, absolute preemption is executed. Interruptible tasks are immediately suspended and the breakpoint is saved. Non-interruptible tasks are set with a maximum waiting timeout of 200ms. If the timeout occurs, the task is terminated, rolled back and marked for retransmission. The emotion linkage and tool bypass module 304 is used to integrate text semantics and device behavior data to calculate emotion confidence, link screen, vibration and TTS parameters according to three-level thresholds, and align multimodal commands uniformly with audio timestamps; whitelisted tools have independent bypass routing, matching only the highest priority tools in a single round, local control tools are directly issued and executed, cloud query tools are set with dynamic timeout, and the timeout loads fallback dialogue without blocking the main dialogue. The Style Security Integrated Management Module 305 is used to solidify fixed personas that cannot be modified. User preferences within a session are temporarily cached as style increments and take effect immediately. After the session ends, they are grouped by dimension, and conflicts are merged using both priority and timestamp rules. The system is asynchronously persisted and retained for traceability to support version rollback. The principle of least privilege is followed throughout the process. Devices, networks, and tools are all whitelisted, and hot updates only distribute configurations without loading executable code.

[0065] A control device for an ESP32-S3 dual-path low-latency multimodal AI toy equipped with a streaming large model is provided. The device is used to execute executable instructions to perform the aforementioned control method for the ESP32-S3 dual-path low-latency multimodal AI toy equipped with a streaming large model.

[0066] A computing device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute any one of the ESP32-S3 cold and hot dual-path low-latency multimodal AI toy control methods equipped with a streaming large model.

[0067] The methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.

[0068] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0069] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application.

Claims

1. A method for controlling ESP32-S3 dual-path low-latency multimodal AI toys equipped with a streaming large-scale model, characterized in that, include: Multimodal signals are acquired based on ESP32-S3, and wake-up detection, VAD audio framing, and fixed-point energy calculation are performed locally. The camera stops frame extraction by default, and only triggers a single frame extraction by an event. During the interaction period, it extracts frames at a low frequency of seconds, and uploads them after cropping and compressing them by ROI. It then goes into sleep mode after being left idle for 60 seconds. Power consumption is reduced through time-sharing task scheduling, peripheral power-off, and dynamic voltage regulation. The data is encapsulated into four-segment binary event packets, with each packet payload not exceeding 2KB. It is uploaded with double encryption and accompanied by CRC integrity verification. The cloud-based assembly of fixed personas, short conversation history, in-conversation style, and device status constitutes the minimum context. A global hard limit of 900 tokens and priority pruning rules are set. Short history is retained for a maximum of 7 rounds and a 120s time-series expiration mechanism is set. The large model streaming interface is called. The first fragment is sent immediately after the length and semantics are both determined, with a 400ms fallback timeout. Subsequent fragments are transmitted at adaptive intervals to stabilize the first character response latency. Non-real-time tasks, including style merging and log storage, are classified into cold paths and managed using a FIFO main queue, with tasks of the same enqueue time sorted by priority. Scheduling is only performed when the CPU, memory, and network loads all reach the idle threshold simultaneously; absolute preemption is executed when a hot path is triggered; interruptible tasks are immediately suspended and breakpoints are saved; non-interruptible tasks are set to a maximum waiting timeout of 200ms, and if the timeout is exceeded, the task is terminated, rolled back, and marked for retransmission. Emotion confidence is calculated by integrating text semantics and device behavior data, and screen, vibration and TTS parameters are linked in a three-level threshold manner. The instructions are uniformly aligned with audio timestamps. Whitelisted tools are routed independently, and only the highest priority tools are matched in a single round. Local control tools are directly executed, while cloud query tools are set with dynamic timeouts. Timeout loading of fallback phrases does not block the main dialogue. Fixed personas are fixed and cannot be modified. User preferences within a session are temporarily cached as style increments and take effect immediately. After the session ends, users are grouped by dimension, and conflicts are merged according to both priority and timestamp rules. The data is asynchronously persisted and retained for traceability to support version rollback. The principle of least privilege is followed throughout the process. Devices, networks, and tools are all whitelisted. Hot updates only distribute configurations and do not load executable code.

2. The ESP32-S3 dual-path low-latency multimodal AI toy control method with a large streaming model as described in claim 1, characterized in that, The cloud-based assembly of a fixed persona, short conversation history, in-conversation style, and device status constitutes the minimum context. A global hard cap of 900 tokens and priority pruning rules are set. Short history is retained for a maximum of 7 rounds with a 120-second expiration mechanism. The large model streaming interface is called, and the first fragment is immediately sent after both length and semantic checks are met, with a 400ms fallback timeout. Subsequent fragments are transmitted at adaptive intervals, stabilizing the first-word response latency. This includes: In response to the low-latency chat interaction needs of toys, a dual constraint of independent token upper limit and short history expiration is introduced. Threshold pre-configuration is implemented for four types of context components: fixed persona, short conversation history, conversation style, and structured device status. Adaptive rules for dynamically adding and deleting modules according to different interaction scenarios are provided. A simplified tool description of ≤80 tokens is temporarily added for tool call scenarios and automatically removed after execution. Based on the optimization goals of a global hard cap of 900 tokens and no loss of core interactive information, the context content is pruned with a four-level priority decreasing pruning. After pruning, the total length is re-verified, and a minimal context sequence with total quantity control, hierarchical pruning, and scenario adaptation is constructed. The expired short history after cleaning is transferred to the cold path and asynchronously written to the log storage, so as not to lose complete interactive data. First, the assembled minimum context is sent to the large model streaming inference interface, then the validity of the first segment is checked. The first packet is sent and the mapping is completed based on the length threshold and semantic integrity dual judgment. The utility class reply is adapted to the dedicated judgment rules, and a 400ms fallback timeout forced sending mechanism is provided. The semantic priority segmentation rule of 8~32Token and the three-level adaptive interval adjustment logic are determined. The timing of text and audio segments is bound, and low-latency streaming output control information adapted to the ESP32-S3 embedded terminal is generated.

3. The ESP32-S3 dual-path low-latency multimodal AI toy control method with a large streaming model as described in claim 1, characterized in that, Non-real-time tasks, including style merging and log storage, are classified into cold paths and managed using a FIFO main queue, with tasks of the same enqueue time sorted by priority. Scheduling is only implemented when CPU, memory, and network loads simultaneously reach the idle threshold; absolute preemption is executed when a hot path is triggered; interruptible tasks are immediately suspended and breakpoints are saved; non-interruptible tasks have a maximum waiting timeout of 200ms, after which they are terminated, rolled back, and marked for retransmission, including: In response to the scheduling requirements of AI toy hot path low latency priority, a dual constraint of task priority classification and resource load threshold is introduced. Enqueue marking and lifecycle management are implemented for non-real-time tasks including style merging, log storage, and device diagnosis. Only two types of high-value tasks, namely style merging at the end of the session and fault diagnosis, are allowed to skip queue. Based on the optimization goals of zero blocking of hot links and controllable background resources, cold tasks are classified and screened, and a standardized cold task scheduling sequence with FIFO main queue sorting, priority queueing for limited categories, and automatic destruction of expired or overdue tasks is constructed. Five types of destruction rules are provided: expiration, session expiration, resource overdue, deduplication, and preemption timeout. First, verify that the CPU, memory, and network loads are all within acceptable limits and there is no active hot interaction. Then, start the parallel scheduling of cold tasks. Rely on the absolute preemption mechanism of hot paths to complete the resource conflict mapping. Adapt the handling logic of interruptible task breakpoint suspension and non-interruptible task waiting for safe interruption point. Determine the rules for 200ms timeout termination rollback and waiting for retransmission marking. Match the multi-layer parallel resource upper limit constraints and generate cold and hot path collaborative scheduling implementation information adapted to ESP32-S3 toy terminal.

4. The ESP32-S3 dual-path low-latency multimodal AI toy control method with a large streaming model as described in claim 1, characterized in that, Emotion confidence is calculated by weighting text semantics and device behavior data, and screen, vibration, and TTS parameters are linked in a three-level threshold manner, with audio timestamps used to unify command alignment; whitelisted tools are routed independently, matching only the highest priority tools in a single round, with local control tools executed directly, and cloud query tools having dynamic timeouts. Timeout loading of fallback phrases does not block the main dialogue, including: Combining the optimization requirements of multimodal anthropomorphic interaction of AI toys and low latency of the main link, we introduce dual constraints of text 0.7 / behavior 0.3 dual-branch weighted fusion and tool whitelist classification control. We preprocess and prioritize six types of emotional features and tool request intent, and abandon the general agent multi-round chain call mode. Based on the goals of emotional confidence level adaptation and zero-blocking main dialogue for tool calls, the emotional linkage strategy and tool execution path are graded and filtered, and a standardized scheduling sequence with three-level confidence thresholds and single-round highest priority matching is constructed. The priority of emotional conflict and the classification and execution rules of local / cloud tools are clarified. Emotion switching waits for the end of the current audio sentence to avoid parameter jumps within the same line. First, using the TTS audio timestamp as a global benchmark, the screen animation and vibration commands are aligned through three layers of timing: preloading, synchronous execution, and termination. Then, the tool intent is started in an independent bypass route. Based on the emotion grading rules, multimodal parameter quantification and mapping are completed. The handling logic of three levels of dynamic timeout is adapted to local tool direct execution and cloud query. The timeout fallback dialogue and abnormal degradation rules are determined. The tool results are only used as supplementary content to the dialogue and are not embedded in the main inference link. The emotion-tool collaborative scheduling implementation information adapted to the ESP32-S3 toy terminal is generated.

5. The ESP32-S3 dual-path low-latency multimodal AI toy control method with a large streaming model as described in claim 4, characterized in that, Fixed personas are fixed and cannot be modified. User preferences within a session are temporarily cached as style increments and take effect immediately. After the session ends, users are grouped by dimension, and conflicts are merged according to both priority and timestamp rules. The data is asynchronously persisted and retained for traceability and version rollback support. The principle of least privilege is followed throughout the process. Devices, networks, and tools are all whitelisted. Hot updates only distribute configurations and do not load executable code, including: Combining the need for unified positioning of AI toy characters with the need to consider user personalization and safety compliance in children's scenarios, we introduce layered isolation of character style and dual constraints of minimum permissions in all dimensions. We implement pre-configuration and lifecycle marking for fixed character content, incremental conversation style, and permission boundary rules, and clarify the core constraint that hot updates only distribute whitelist configurations, parameters and templates, and do not load executable code. In accordance with the optimization goals of zero-blocking main link for style persistence and no unauthorized access control throughout the process, style increments and security rules are graded, verified and screened, and a standardized style management sequence is constructed that is grouped by four dimensions, conflicts are judged by priority and timestamp, and traceability, retention and rollback are supported. A security control system with three-level whitelists for devices, networks and tools plus content filtering is also provided. Style version snapshots are retained for a minimum of 90 days and historical version rollback is supported. First, style increments are cached in session memory and applied immediately without being written to the persistent layer. After the session ends, the process is transferred to the cold path for merged persistence. Conflict verification mapping is completed based on rules of mutual exclusion overriding and overlapping fusion. The handling logic of dual-media storage for edge and cloud and configuration signature distribution is adapted. The rules for version snapshot retention and abnormal security degradation are determined. The key is stored in the hardware security zone. All configuration distributions are verified by digital signatures. Style security integrated management and control implementation information adapted to the ESP32-S3 toy terminal is generated.

6. A low-latency, multimodal AI toy control system with a dual-path (cold and hot) flow cytometry model, characterized in that: The system is used to execute executable instructions to perform the ESP32-S3 dual-path low-latency multimodal AI toy control method with a streaming large model as described in any one of claims 1 to 5.

7. A control device for a low-latency, multimodal AI toy with a dual-path (cold and hot) flow cytometry model, characterized in that, The device is used to execute executable instructions to perform the ESP32-S3 dual-path low-latency multimodal AI toy control method with a streaming large model as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, include: First processor; And a memory for storing executable instructions of the first processor; wherein the first processor is configured to execute the ESP32-S3 cold and hot dual-path low-latency multimodal AI toy control method according to any one of claims 1 to 5 by executing the executable instructions.

9. A computing device, the device comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein, When the computer program instructions are executed by the processor, the device is triggered to execute the ESP32-S3 cold and hot dual-path low-latency multimodal AI toy control method with a streaming large model as described in any one of claims 1 to 5.

Images