Desktop accompanying robot system based on double ESP32 architectures and interaction control method

Abstract

The invention relates to the technical field of intelligent robots and human-computer interaction, and provides a desktop accompanying robot system based on double ESP32 architectures and an interaction control method. According to the system, touch, voice, images, environment states and other multi-mode interaction data are collected through a sensing layer, a decision-making layer carries out unified preprocessing on the multi-mode data and carries out communication interaction with a cloud server, semantic analysis is completed at a cloud end, and a standardized interaction control instruction is generated. Based on the interaction control instruction, the system realizes expression display, motion control, voice playing and other multi-mode collaborative responses through a double ESP32 division collaborative architecture, and forms closed-loop control through execution state feedback. According to the system, multi-modal interaction input and a cloud semantic understanding result are unified and integrated, time sequence conflicts and response delay caused by decentralized processing are avoided, and low-delay and stable multi-modal interaction output is realized, so that the interaction continuity, the operation stability and the user experience of the desktop accompanying robot are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot and human-computer interaction technology, specifically to a desktop companion robot system and interactive control method based on a dual ESP32 architecture. Background Technology

[0002] With the improvement of embedded hardware performance and the development of artificial intelligence technology, desktop companion robots, as intelligent terminals integrating voice interaction, facial expression display, and motion feedback, are gradually being applied to scenarios such as family companionship, emotional interaction, educational assistance, and intelligent assistants. These robots typically need to perceive user input through multiple methods such as touch, voice, and images, and provide timely, coherent, and emotionally expressive feedback based on the perception results, thereby enhancing the naturalness and immersion of human-computer interaction.

[0003] Existing desktop companion robot systems mostly adopt a single main control chip architecture, with the same control unit simultaneously handling tasks such as sensor data acquisition, voice processing, network communication, and hardware driving of servos and displays. In practical applications, this type of architecture is prone to main control resource contention when facing concurrent processing of multimodal data, leading to system response delays, asynchronous interactive actions, or decreased operational stability, making it difficult to meet the real-time and continuous requirements of natural interaction.

[0004] Furthermore, existing multimodal interaction technologies often revolve around a single trigger event, such as responding only to voice wake-up or a single touch event. There is a lack of unified interaction state modeling and collaborative processing mechanisms among various types of perceptual information. When multiple interaction methods such as touch, voice, and images occur simultaneously or consecutively, the system usually adopts a decentralized processing approach, triggering the response of the corresponding modules separately. This can easily lead to inconsistencies in the timing between voice, facial expressions, and actions, affecting the interactive experience.

[0005] In the application of cloud-based intelligent services, existing technologies typically treat services such as speech recognition, semantic understanding, and speech synthesis as independent processes, with the results often being sent to local devices in batches or asynchronously. The local control system needs to integrate the results from multiple sources, which not only increases the complexity of the system implementation but also easily leads to command conflicts or accumulated delays in the event of network fluctuations or inconsistent service responses, further affecting the consistency of interaction.

[0006] Meanwhile, existing systems often employ simple event-triggered control methods to address the common needs for facial expression display and motion control in desktop companion robots, lacking unified management of the overall state of the interaction process. When users frequently trigger interaction events or the environmental state changes, the system struggles to promptly determine the continuity and validity of the interaction intent, easily leading to accidental touch responses, repetitive actions, or invalid executions, increasing hardware load and shortening device lifespan.

[0007] Therefore, how to achieve efficient processing and collaborative decision-making of multimodal perception data on resource-constrained embedded platforms, avoid resource contention issues caused by single-board architecture, and achieve unified and coherent output of voice, facial expressions, and actions while ensuring low latency has become a pressing technical problem to be solved in the field of desktop companion robots. Summary of the Invention

[0008] To address the problems of system resource competition, asynchronous interaction responses, and insufficient operational stability in existing desktop companion robots during multimodal interaction, this invention proposes a desktop companion robot system and interaction control method based on a dual ESP32 architecture. This technical solution separates the deployment of perception, decision-making, and execution functions, and combines a collaborative mechanism of cloud-based semantic parsing and local real-time control to achieve unified processing and coordinated response to multimodal interaction information such as touch, voice, and images. This improves the real-time performance, consistency, and stability of the interaction response on resource-constrained embedded platforms.

[0009] To address this, a desktop companion robot system based on a dual ESP32 architecture is proposed, characterized in that the system comprises: The perception layer is used to collect user interaction data and environmental status data; The decision layer includes a first ESP32 main control chip. The decision layer communicates with the perception layer and processes the data collected by the perception layer, and generates interactive control commands based on the processing results. The execution layer includes a second ESP32 main control chip. The execution layer communicates with the decision layer and is used to receive the interactive control commands and drive the actuator to complete the corresponding interactive actions. A cloud server communicates with the decision-making layer to perform semantic parsing on the data uploaded by the decision-making layer and generate the interactive control instructions. The decision-making layer and the execution layer achieve functional separation of perception decision-making and hardware execution through a dual ESP32 architecture, and form a closed-loop control through instruction issuance and status feedback to improve the real-time performance and operational stability of the desktop companion robot's interaction process.

[0010] The decision-making layer and the execution layer are connected via UART serial port communication to transmit interactive control commands and receive execution status feedback information.

[0011] The perception layer includes an image acquisition module, an audio acquisition module, a touch sensing module, and an environmental perception module. The multimodal interaction data acquired by the perception layer is uniformly transmitted to the decision layer for processing.

[0012] The image acquisition module is connected to the decision layer via an SPI interface, the audio acquisition module is connected to the decision layer via an I2S interface, and the touch sensing module and the environmental perception module are connected to the decision layer via an I2C interface.

[0013] The execution layer includes a servo drive module, a display drive module, and a power management module. The execution layer drives the servo drive module, the display drive module, and the power management module to perform corresponding operations according to the interactive control commands.

[0014] The display driver module drives at least two circular displays via the SPI interface to play facial animations.

[0015] The display driver module adopts a dual-buffering display mechanism, which completes the drawing of facial animation frame data in the background before outputting it to the circular display screen to reduce screen tearing.

[0016] The animated expressions are GIF files that have been adapted, including resolution scaling, display area cropping, and format conversion, to match the display characteristics of the circular display screen.

[0017] The power management module is used to detect the robot's operating status and charging status, and to perform sleep control and charging protection operations based on the detection results.

[0018] A desktop companion robot interactive control method based on a dual ESP32 architecture, the method comprising the following steps: S01 collects at least one type of user interaction data through the perception layer, the interaction data including voice data, touch event data or image data; S02 transmits the interactive data to the decision layer based on the first ESP32 main control chip, processes the interactive data, and then uploads it to the cloud server; S03 The cloud server performs semantic parsing on the interactive data. During the semantic parsing process: a) When the interaction data includes touch event data, generate corresponding interaction control instructions based on the touch event type; b) When the interactive data includes voice data, interactive control commands related to voice interaction are generated through speech recognition and speech synthesis processing; c) When the interactive data includes image data, generate face location or target location information based on the image data, and generate interactive control commands for driving the actuator to adjust its position; S04 returns the generated interactive control instructions to the decision layer, which then parses the interactive control instructions and sends the real-time execution instructions to the execution layer based on the second ESP32 main control chip. S05 The execution layer drives the execution mechanism to complete the corresponding interactive action according to the real-time execution instruction, and feeds back the execution status to the decision layer; The decision-making layer described in S06 adjusts subsequent interactive control commands based on the execution status feedback results to form a closed-loop control process for multimodal interaction.

[0019] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention constructs a unified and stable interactive control system for desktop companion robots by using multimodal interaction states as a unified state representation of the system, combining cloud-based semantic parsing with local real-time control, and employing a dual ESP32 collaborative architecture for execution and decision-making. The multimodal interaction states are no longer treated as independent processing objects of single sensor signals or single interactive events, but rather as a unified interactive state space encompassing touch events, voice input, image information, and environmental states. Data from touch sensors, microphone arrays, cameras, and environmental perception modules are centrally aggregated at the decision layer for unified preprocessing and state modeling, enabling the system to form a comprehensive understanding of the current interactive scenario at the overall level. This avoids the problems of fragmented interaction, inconsistent responses, or conflicting actions caused by the independent triggering and processing of various sensor modules in existing technologies.

[0020] 2. This invention no longer treats cloud-based intelligent services merely as simple speech transcription or response generation tools, but rather integrates cloud-based semantic analysis results as a crucial component of multimodal interactive control decision-making. After uploading touch events and real-time voice data to the cloud, the cloud server performs unified semantic understanding of the multimodal input. Based on speech recognition, large-scale model inference, and speech synthesis processing, it integrates voice playback commands, facial expression display commands, and motion control commands and sends them all at once to the local decision-making layer. This centralized command generation method avoids the timing inconsistencies caused by separate service returns and dispersed command distribution in existing technologies, achieving a shift from "distributed interactive response" to "unified semantic-driven multimodal collaborative response."

[0021] 3. This invention employs a dual ESP32 collaborative hardware architecture, deploying perception and decision-making functions on the host computer and hardware driver and state management functions on the slave computer, forming a closed-loop collaborative control mechanism through serial communication. This architecture no longer relies on a single main control chip to simultaneously handle perception data acquisition, cloud communication, and hardware driver tasks. Instead, it significantly reduces the risk of system resource contention and improves overall response speed and operational stability through functional decoupling. The decision-making layer generates standardized execution instructions under unified interactive state semantics, enabling the execution layer to complete actions such as servo movement, facial expression display, and power management under clear instruction constraints. This avoids the stuttering, action delay, or interaction distortion caused by task preemption in traditional single-board architectures.

[0022] Meanwhile, the multimodal interaction state possesses temporal continuity and event correlation characteristics, enabling the system to respond based on the overall evolution of the interaction process, rather than relying solely on discrete triggering judgments based on instantaneous input. Compared to interaction methods based on simple thresholds or single-event triggers, this invention can more smoothly organize the coordinated output of facial expressions, actions, and speech, improving the naturalness and consistency of the interaction process.

[0023] 4. This invention introduces a unified state modeling and instruction integration mechanism during the interactive control process, enabling the system to effectively distinguish between stable interactive behaviors and occasional disturbance inputs. When touch, voice, or environmental perception data exhibits short-term fluctuations, noise interference, or incomplete information, the system can determine the credibility of the interactive intent based on the overall interactive state, avoiding unnecessary action responses caused by accidental touches, misidentifications, or momentary anomalies. This reduces invalid interactions and the number of hardware executions, thereby lowering equipment wear and tear.

[0024] When interaction conditions are insufficient or the environment changes significantly, the system can constrain or delay the execution of instructions through unified decision-making logic, thereby improving the safety and reliability of the robot's operation in complex environments. By incorporating multimodal inputs, cloud-based semantic understanding results, and local execution status into a unified closed loop, the system achieves controllable management of interaction risks and improves the interpretability and predictability of the interaction decision-making process.

[0025] Through the aforementioned closed-loop mechanism, this invention organically combines sensory data acquisition, multimodal interaction state modeling, cloud-based semantic parsing, command integration and distribution, execution-layer action feedback, and operational status management to construct a complete multimodal interaction closed-loop control system. The status feedback information acquired after interaction execution is not only used to confirm the current interaction process but also for continuous optimization of subsequent interaction logic and parameters, enabling the system to adapt to different usage scenarios, user habits, and changes in operating conditions. Compared to existing technologies, this invention exhibits higher stability, a more natural interactive experience, and superior system reliability during long-term operation, significantly enhancing the practical value and user experience of desktop companion robots. Attached Figure Description

[0026] Figure 1 This is a top-level system architecture diagram of a desktop companion robot system based on a dual ESP32 architecture; Figure 2 This is a hardware architecture diagram of a desktop companion robot system based on a dual ESP32 architecture. Figure 3 This is a timing diagram illustrating a multimodal collaborative interaction of a desktop companion robot system based on a dual ESP32 architecture. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0028] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0029] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. Example

[0030] The following detailed description, in conjunction with the accompanying drawings, illustrates a specific implementation of the desktop companion robot system and interactive control method based on a dual ESP32 architecture provided by the present invention. This embodiment is merely for illustrating the technical solution of the present invention and does not constitute a limitation on the scope of protection of the present invention.

[0031] The overall architecture of the desktop companion robot system described in this embodiment is as follows: Figure 1 As shown, the system consists of five parts: perception layer, decision layer, execution layer, cloud layer, and debugging layer. Each layer operates independently and achieves collaborative linkage through specific communication links, thus forming a complete collaborative interactive desktop companion robot system.

[0032] The perception layer, as the system's data acquisition terminal, collects user interaction data and environmental status data, forming the foundation for multimodal interaction. The decision layer, as the system's decision-making center, receives data uploaded from the perception layer, preprocesses multimodal data, performs logical judgments and generates instructions, communicates with the cloud layer, and simultaneously sends real-time execution instructions to the execution layer, achieving a closed loop of "data reception—logical judgment—instruction issuance." The execution layer, as the system's hardware execution terminal, receives instructions from the decision layer and drives actuators such as servos, displays, and power modules to complete specific actions, serving as the core carrier for transforming decision instructions into actual physical behavior. The cloud layer, deployed on a public cloud platform, integrates speech recognition, large-scale model inference, and speech synthesis services. It performs semantic understanding, personalized response generation, and standardized instruction conversion on the multimodal data uploaded from the decision layer, providing the system with intelligent interaction capabilities. The debugging layer is an auxiliary development module, mainly including a 4-mic sound source localization module and a PC-side face detection service. It is only activated during system development and debugging to calibrate the data acquisition accuracy of the perception layer and optimize the accuracy of the decision layer's logical operations, and does not participate in system operation under normal care scenarios.

[0033] In this embodiment, the perception layer includes an image acquisition module, an audio acquisition module, a touch sensing module, and an environmental perception module. The image acquisition module is a camera connected to the decision layer via an SPI interface. The decision layer controls it to periodically acquire environmental image data at a preset frequency, such as one frame every 1 to 2 seconds. After acquisition, the decision layer performs preliminary preprocessing such as scaling and compression on the image data. The audio acquisition module is a 4-mic microphone array connected to the decision layer via an I2S interface. It is used to acquire user voice data, and the decision layer performs noise reduction and other preprocessing on the voice data after acquisition. The touch sensing module is an MPR121 touch sensor connected to the decision layer via an I2C interface. It supports multi-channel touch configuration and can define the touch position corresponding to each channel according to the robot's structure. When a touch duration is detected to be greater than or equal to a preset threshold (e.g., 200ms), stable and valid touch event data is generated. The environmental perception module includes a gyroscope and a laser rangefinder, also connected to the decision layer via an I2C interface. It is used to acquire the robot's posture data and distance data of the surrounding environment, providing environmental data support for functions such as fall detection, automatic obstacle avoidance, and sleep determination. The raw data acquired by the above perception layers is transmitted to the decision layer in real time through the corresponding interfaces.

[0034] In this embodiment, the decision-making layer uses the ESP32 S3 main control chip as the host computer, and its hardware structure and interface connection relationship are as follows: Figure 2As shown. The ESP32 S3 host computer integrates a Wi-Fi communication module, and deploys interfaces corresponding to each acquisition module in the perception layer around the main control chip. Its core functions are centralized preprocessing of perception data, cloud communication interaction, logical decision-making, and generation of execution instructions. The ESP32 S3 host computer connects to the camera module via the SPI interface. After preliminary preprocessing of the acquired image data, when face tracking is required, it uploads the image data to the PC-side face detection service in the debugging layer via an HTTP communication link. It receives the face coordinate information returned by this service and calculates the face's position offset relative to the robot based on the coordinate data, generating subsequent face tracking control logic commands to drive the servo motors in the execution layer. The ESP32 S3 host computer connects to a 4-mic microphone array via the I2S interface. After noise reduction processing of the acquired voice data, it uploads it to the cloud-layer speech recognition service via an HTTP communication link. During system debugging, this interface can also connect to the sound source localization module in the debugging layer to calibrate audio acquisition accuracy. The ESP32 S3 host computer connects to the MPR121 touch sensor via the I2C interface. When a valid touch event is detected, the touch event data is packaged and uploaded along with real-time voice data to the cloud-layer server, realizing multimodal input of touch and voice. The ESP32 S3 host computer connects to the I2C... The interface connects to a gyroscope and a laser rangefinder to collect attitude and environmental information, ensuring system stability. The ESP32 S3 host computer establishes a wired communication link with the ESP32 P4 slave computer in the execution layer via a UART serial port (e.g., with a baud rate of 115200) to send real-time execution commands such as motion control and facial expression display to the slave computer. Simultaneously, it receives feedback data on servo status, display status, and power status from the slave computer, enabling real-time monitoring of the execution layer. The ESP32 S3 host computer provides stable power to the relevant acquisition modules in the perception layer through its own power module and collaborates with the power management module in the execution layer to achieve coordinated management of the overall system power supply status.

[0035] In this embodiment, the execution layer also adopts a dual-board architecture, and its hardware structure and the collaborative relationship with the decision-making layer are as follows: Figure 2As shown, the execution layer uses the ESP32 P4 main control chip as the lower-level machine, and deploys servo drive module, display drive module, storage module, wireless charging module and power management module around the main control chip. Its core function is real-time response and status management of hardware drivers. The ESP32 P4 lower-level machine connects to the PCA9685 servo drive module via the I2C interface to drive multiple servos to execute motion control commands. The servo control logic can include smooth periodic motion control based on a first-order sine curve and non-periodic motion control based on keyframes. The upper-level machine sends motion control commands with action identifiers via the UART serial port. The lower-level machine matches the corresponding motion parameters according to a predefined motion library and drives the servos to execute the corresponding actions. The ESP32 P4 lower-level machine connects to a 240×240 resolution circular dual-screen via the SPI interface, driving GIF emoticon display based on the LVGL framework. It supports symmetrical display for both eyes or independent display for both eyes. In the symmetrical display scenario, only one side's GIF file needs to be provided; the other side is generated by horizontal flipping using software algorithms, thus simplifying the emoticon adaptation process. The ESP32 P4 lower-level machine connects to an SD card via the SPI interface to store the adapted GIF emoticon files. The GIF files can be converted to version 89a, and the single file size is controlled within a preset range to ensure loading and playback efficiency. The card can also be used to store motion parameters and operation logs; the ESP32 P4 lower-level machine connects to the wireless charging module via the I2C interface and works in conjunction with the power management module. When it detects that the robot is placed on the wireless charging base, it automatically triggers the charging process and enters sleep or wake-up mode according to the operating status; the power management module is responsible for switching between the lithium battery and the external power adapter, and monitors the charging status and temperature status in real time. When it detects that there has been no interaction for a long time, it triggers the sleep mode to reduce power consumption. When it detects abnormal conditions such as overcharging or overheating, it automatically cuts off the charging link and feeds back the power supply status and charging progress to the upper-level machine.

[0036] In the aforementioned dual ESP32 architecture, the decision-making layer and the execution layer form a closed-loop control mechanism of "command issuance - status feedback" through the UART serial port. The host computer focuses on sensing data processing, cloud interaction, and logical decision-making, while the slave computer focuses on hardware driving and status monitoring. This avoids the system lag and response delay problems caused by the simultaneous occupation of main control resources by sensing data acquisition, cloud communication, and hardware driving tasks in the single-board architecture, and significantly improves the system's response speed and operational stability.

[0037] Based on the above system structure, this embodiment further illustrates a collaborative interactive control method, the timing flow of which is shown in the example below. Figure 3As shown. Taking multimodal interaction triggered by touch events as an example, when a user touches the robot's nose area, the MPR121 touch sensor collects a stable touch signal and generates touch event data; after receiving the touch event, the ESP32 S3 host computer synchronously collects real-time voice data within about 20ms and uploads the touch event data and voice data to the cloud server via HTTP protocol; after receiving the data, the cloud server first calls the speech recognition service to convert the voice data into text information, then calls the large model inference service to generate semantic response text by combining the type of touch event and the voice text content, and then calls the speech synthesis service to convert the semantic response text into voice audio data; after completing the above processing, the cloud server integrates the voice playback command, expression display command, and motion control command, and sends them to the ESP32 S3 host computer at once via HTTP protocol; after receiving the integrated command at about 205ms, the ESP32 S3 host computer parses the command and forwards the expression display command and motion control command to the ESP32 P4 slave computer via UART serial port; the ESP32 P4 slave computer receives the integrated command at about 210ms. The system continuously drives the servo motor to complete the head-turning action and displays the corresponding facial expression on both screens. Simultaneously, the host computer controls the speaker to play audio, thus completing a full closed loop from touch event triggering to multimodal interactive action execution within approximately 215ms. Actual testing has verified that the overall response time of this multimodal collaborative interaction process can be controlled within 220ms, meeting the real-time requirements of natural interaction.

[0038] As can be seen from the above embodiments, the present invention effectively decouples the perception, decision-making and execution, and driving functions through a dual ESP32 collaborative architecture, and combines cloud-based multimodal semantic parsing capabilities to achieve a low-latency, high-stability multimodal collaborative interactive desktop companion robot system, significantly improving the system's operational stability, response speed, and companion interaction experience. Example

[0039] Based on the desktop companion robot system described in Embodiment 1, this embodiment further provides a multimodal interaction control method.

[0040] After the desktop companion robot is powered on, the execution layer first performs a power status check and feeds back the battery power and charging status information to the decision layer. Once the decision layer confirms that the system is operational, it initializes the modules of the perception layer and establishes a communication connection with the cloud server, putting the system into a standby listening state.

[0041] When a user interacts with the robot, the system triggers a multimodal data acquisition process in an event-driven manner.

[0042] Taking touch triggering as an example, when the touch sensing module detects a user's touch operation on a designated area of ​​the robot and the touch duration exceeds a preset threshold, it generates touch event data and sends it to the decision layer. After receiving the touch event data, the decision layer records the corresponding timestamp information and simultaneously triggers the audio acquisition module to start collecting user voice data; in some implementations, the decision layer also controls the image acquisition module to acquire current environmental image data.

[0043] The decision-making level performs local preprocessing on the collected voice data, touch event data, and image data, and encapsulates the preprocessed multimodal interaction data into a unified format data packet, which is then uploaded to the cloud server via network communication.

[0044] After receiving the multimodal interaction data, the cloud server sequentially executes the semantic parsing processing flow: When the data packet contains voice data, the voice recognition service is invoked to convert the voice data into text information; then, the text information is combined with the touch event data, and the large model inference service is invoked to perform semantic parsing of the user's intent; in the presence of image data, the image data is parsed to obtain face location or target location information.

[0045] After semantic parsing is completed, the cloud server generates interactive control instructions based on the parsing results. These interactive control instructions include facial expression display instructions, motion control instructions, and voice playback instructions.

[0046] The cloud server sends the generated interactive control commands to the decision-making layer. After receiving the interactive control commands, the decision-making layer parses and integrates the commands, and then sends the real-time execution commands to the execution layer in one go via serial communication.

[0047] After receiving a real-time execution instruction, the execution layer drives the corresponding execution module to complete the interactive action according to the instruction content.

[0048] When a motion control command is received, the execution layer drives the servo to complete the head turning or attitude adjustment action; when an expression display command is received, the execution layer reads the corresponding expression animation file from the local storage medium and plays the expression animation on the circular display screen through the display driver module; at the same time, the execution layer performs voice playback related operations as needed.

[0049] After completing the interactive action, the execution layer collects the servo motor's operating status, display status, and power status information, and feeds the execution status back to the decision layer.

[0050] After receiving the execution status information from the execution layer, the decision layer analyzes and judges the execution status. When the execution status is consistent with the expectation, the current interaction process is completed. When a deviation in the execution status is detected, the decision layer adjusts the subsequent interaction control commands, thereby forming a closed-loop control process for multimodal interaction.

[0051] Through the above process, the system can complete a multimodal interactive response from the triggering of an interactive event to the execution of an action within approximately 200 milliseconds.

Claims

1. A desktop companion robot system based on a dual ESP32 architecture, characterized in that, The system includes: The perception layer is used to collect user interaction data and environmental status data; The decision layer includes a first ESP32 main control chip. The decision layer communicates with the perception layer and processes the data collected by the perception layer, and generates interactive control commands based on the processing results. The execution layer includes a second ESP32 main control chip. The execution layer communicates with the decision layer and is used to receive the interactive control commands and drive the actuator to complete the corresponding interactive actions. A cloud server communicates with the decision-making layer to perform semantic parsing on the data uploaded by the decision-making layer and generate the interactive control instructions. The decision-making layer and the execution layer achieve functional separation of perception decision-making and hardware execution through a dual ESP32 architecture, and form a closed-loop control through instruction issuance and status feedback to improve the real-time performance and operational stability of the desktop companion robot's interaction process.

2. The desktop companion robot system based on a dual ESP32 architecture according to claim 1, characterized in that, The decision-making layer and the execution layer are connected via UART serial communication to transmit interactive control commands and receive execution status feedback information.

3. The desktop companion robot system based on a dual ESP32 architecture according to claim 1, characterized in that, The perception layer includes an image acquisition module, an audio acquisition module, a touch sensing module, and an environmental perception module. The multimodal interaction data acquired by the perception layer is uniformly transmitted to the decision layer for processing.

4. The desktop companion robot system based on a dual ESP32 architecture according to claim 3, characterized in that, The image acquisition module is connected to the decision layer via an SPI interface, the audio acquisition module is connected to the decision layer via an I2S interface, and the touch sensing module and the environmental perception module are connected to the decision layer via an I2C interface.

5. The desktop companion robot system based on a dual ESP32 architecture according to claim 1, characterized in that, The execution layer includes a servo drive module, a display drive module, and a power management module. The execution layer drives the servo drive module, the display drive module, and the power management module to perform corresponding operations according to the interactive control commands.

6. The desktop companion robot system based on a dual ESP32 architecture according to claim 5, characterized in that, The display driver module drives at least two circular displays via an SPI interface to play facial animations.

7. The desktop companion robot system based on a dual ESP32 architecture according to claim 6, characterized in that, The display driver module adopts a double buffering display mechanism, which completes the drawing of facial animation frame data in the background before outputting it to the circular display screen to reduce screen tearing.

8. The desktop companion robot system based on a dual ESP32 architecture according to claim 7, characterized in that, The animated expressions are GIF files that have been adapted. The adaptation process includes resolution scaling, display area cropping, and format conversion to match the display characteristics of the circular display screen.

9. The desktop companion robot system based on a dual ESP32 architecture according to claim 5, characterized in that, The power management module is used to detect the robot's operating status and charging status, and to perform sleep control and charging protection operations based on the detection results.

10. A desktop companion robot interactive control method based on a dual ESP32 architecture, characterized in that, The method includes the following steps: The user's interaction data is collected through the perception layer, including voice data, touch event data, or image data. The interactive data is transmitted to the decision layer based on the first ESP32 main control chip, and after processing, it is uploaded to the cloud server. The cloud server performs semantic parsing on the interactive data. During the semantic parsing process, when the interactive data includes touch event data, corresponding interactive control instructions are generated by combining the touch event type. When the interactive data includes voice data, interactive control commands related to voice interaction are generated through speech recognition and speech synthesis processing. When the interactive data includes image data, face location or target location information is generated based on the image data, and interactive control commands for driving the actuator to adjust its position are generated. The generated interactive control commands are returned to the decision layer, which parses the interactive control commands and sends the real-time execution commands therein to the execution layer based on the second ESP32 main control chip. The execution layer drives the execution mechanism to complete the corresponding interactive action according to the real-time execution instruction, and feeds back the execution status to the decision layer; The decision-making layer adjusts subsequent interactive control commands based on the execution status feedback results to form a closed-loop control process for multimodal interaction.

Images