Body-equipped agent structure for realizing man-machine conversation based on large model

Abstract

The invention discloses an intelligent body structure for realizing man-machine conversation based on a large model, and the intelligent body structure comprises an intelligent body structure body which can autonomously move and operate outside; and the large language model is deployed on a central processing unit in the body structure and is used for understanding a user language instruction and driving the body structure to act. The large language model understanding instruction comprises the steps that a user sends out an instruction through voice, the instruction is converted into a text instruction after being collected and preprocessed through a voice recognition microphone array, a central processing unit inputs the text instruction into the large language model, the model conducts semantic understanding and intention recognition and generates a decision instruction, and the central processing unit drives the body structure to act in sequence. The method has the beneficial effects that efficient man-machine conversation based on a large model is realized, the intelligent agent with the body can autonomously move and operate, and the man-machine interaction experience and the working efficiency of the intelligent agent are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more particularly to an embodied intelligent body structure for realizing human-computer dialogue based on a large model. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, human-computer dialogue systems have gained widespread application. However, existing human-computer dialogue systems mostly exist in pure software form, lacking embodiment and unable to interact and operate in the physical world like humans. This limits their application in some complex scenarios. For example, in a smart home environment, users expect intelligent agents to not only communicate with them through voice but also autonomously move to specific locations and perform actions based on the conversation, such as turning on the lights in a specific room or adjusting the air conditioning temperature. Furthermore, existing systems have shortcomings in understanding and generating natural language, failing to accurately grasp user intent and generate fluent and accurate responses. Summary of the Invention

[0003] In response to the shortcomings of the existing technology, the purpose of the present invention is to provide an embodied intelligent body structure based on a large model to realize human-computer dialogue. By combining the natural language processing capabilities of the large language model and the physical interaction capabilities of the embodied intelligent body, a more natural, intelligent and practical human-computer dialogue system is realized.

[0004] To achieve the above objectives, the present invention provides the following technical solution: an embodied intelligent agent structure for realizing human-computer dialogue based on a large model, comprising:

[0005] An embodied structure, which is an embodied agent used to move and operate autonomously in the outside world;

[0006] The large language model is deployed on the central processor within the embodied structure to understand the language commands issued by the user and drive the specific structure to move and operate autonomously based on the understood user language commands;

[0007] The specific steps for the large language model to understand language instructions are as follows:

[0008] In step 1, the user issues a command to the embodied structure through voice. The voice recognition microphone array collects and preprocesses the voice signal, and then transmits the preprocessed voice signal to the voice recognition module, which converts it into a text-based command.

[0009] In step 2, the CPU inputs the text instructions into the large language model, which performs semantic understanding and intent recognition on the instructions. Based on different intents, the model generates corresponding decision instructions.

[0010] Step three: The central processing unit drives the embodied structure to move and operate autonomously according to the decision instructions.

[0011] As a further improvement of the present invention, the embodied structure includes:

[0012] The mobile platform is a wheeled or tracked structure equipped with laser radar, cameras, and ultrasonic sensors to sense the surrounding environment and obtain information about obstacles, terrain, and the location of target objects;

[0013] An operating robot arm, mounted on a mobile platform, is used to grasp and manipulate objects;

[0014] A sensor system, comprising a voice recognition microphone array and a touch sensor for receiving user voice commands and sensing tactile information of objects;

[0015] The central processing unit is connected to the mobile platform, sensor system and operating robotic arm to receive voice commands output by the sensor system, perceive tactile information of objects, obtain obstacle information, terrain information and target object position, and control the operating robotic arm to grasp and operate objects.

[0016] As a further improvement of the present invention, the large language model is obtained by first selecting a large language model, and then further fine-tuning the large language model according to the application scenario of the embodied structure.

[0017] As a further improvement of the present invention, the central processing unit in step three drives the embodied structure to move and operate autonomously according to the decision instructions. The specific method is as follows: the central processing unit controls the movement of the mobile platform and the robotic arm according to the decision instructions to realize autonomous navigation, obstacle avoidance, grasping, and operation. In the process of executing the action, the sensor system monitors the environmental changes and the execution of the action in real time, and transmits the feedback information to the central processing unit. The central processing unit adjusts the action parameters according to the feedback information. At the same time, the result of the action execution is fed back to the user in the form of voice or text to inform the user of the completion status of the task.

[0018] As a further improvement of the present invention, the specific way in which the large language model performs semantic understanding of instructions in step 2 is: after receiving a voice instruction, it understands the moving scene instruction in the voice instruction and drives the embodied structure to move to the scene. Then, it receives the next language instruction and understands whether there is a moving scene instruction in it. If so, the moving scene instruction is executed first to drive the embodied structure to move to the scene. If not, it is determined that the language instruction is the current scene instruction, and the operating robot arm is controlled based on the language instruction to grasp or operate the current scene objects.

[0019] As a further improvement of the present invention, the specific steps of the large language model performing semantic understanding of the instruction in step 2 are as follows:

[0020] Step 21: First, complete the question in the language instruction, pass the question along with the context information to the big model, and set a prompt to clearly complete the question;

[0021] Step 2: After completing the question, continue to call the big model and set the intent classification in the prompt. Currently, the design command classification is divided into four types: data query, security anomaly identification, camera control, and chat. After the big model understands the semantics of the question, it uses the Softmax function to perform intent classification probability distribution and selects the intent with the highest score. The Softmax function formula is as follows:

[0022]

[0023] Among them, zk is the score of the kth intent output by the model.

[0024] The beneficial effect of the present invention is that, through the hardware architecture of the embodied intelligent body, the autonomous movement and operation capabilities of the human-computer dialogue system in the physical world are realized, which can meet the diverse needs of users in complex scenarios, such as smart home control, industrial inspection, service robots and other scenarios.

[0025] The use of large language models for natural language processing significantly improves the accuracy of the system's understanding of user commands and the fluency and accuracy of responses, allowing users to communicate with intelligent agents in a more natural and convenient way.

[0026] Optimized model deployment and acceleration technology ensures the system's rapid response during real-time interaction and improves user experience.

[0027] Multi-round dialogue and context understanding mechanisms enhance the naturalness and coherence of human-computer interaction, enabling intelligent agents to better adapt to users' interaction habits and improve interaction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a diagram showing the technical details and process examples of the large language model instruction understanding and decision-making technology of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below with reference to the embodiments shown in the accompanying drawings.

[0030] Reference Figure 1 As shown, the embodiment of the present invention is an embodied intelligent agent structure based on a large model to realize human-computer dialogue, which mainly includes an intelligent agent hardware architecture and a large language model, as follows:

[0031] The hardware architecture of the embodied intelligent body, that is, the embodied structure:

[0032] Mobile Platform: A wheeled or tracked mobile platform equipped with high-precision motors and drives enables autonomous navigation and obstacle avoidance. The mobile platform is equipped with a variety of sensors, such as lidar, cameras, and ultrasonic sensors, to sense the surrounding environment and obtain information about obstacles, terrain, and the location of target objects.

[0033] Manipulator: A multi-degree-of-freedom manipulator is mounted on a mobile platform. Its end features replaceable actuators, such as grippers and suction cups, for grasping and manipulating objects. High-precision motors and encoders are installed at the joints of the manipulator to precisely control its posture and position.

[0034] Sensor system: In addition to sensors on the mobile platform and robotic arm, it is also equipped with a voice recognition microphone array and touch sensors for receiving user voice commands and sensing tactile information from objects. Data from all sensors is transmitted to the central processing unit via a high-speed communication interface.

[0035] Central Processing Unit: It uses a high-performance computing chip with powerful data processing and multi-tasking capabilities. It can process sensor data in real time, run large language model algorithms, and control the movements of mobile platforms and robotic arms.

[0036] Large language model, wherein this embodiment integrates and optimizes the large language model after obtaining it:

[0037] Model Selection and Training: We selected an advanced large language model as the core algorithm. This model, pre-trained on a large amount of natural language data, possesses strong language understanding and generation capabilities. We further fine-tuned the model for embodied agent applications, enabling it to better understand and process language commands related to interacting with the physical world, such as "Please go to the living room" or "Please help me get the remote control off the table."

[0038] Model deployment and acceleration: The trained large language model is deployed on the embodied agent's central processing unit (CPU). To improve the model's speed and response efficiency, the model is optimized using techniques such as model quantization and pruning, reducing the model's computational workload and storage space usage. Furthermore, hardware acceleration technologies such as GPU acceleration and FPGA acceleration are used to further improve the model's inference speed.

[0039] Therefore, under the combination of the embodied structure and the large language model, the embodied intelligent agent structure of this embodiment has a human-computer dialogue process and interaction mechanism:

[0040] Speech Input and Recognition: Users issue commands to the embodied agent through voice. The speech recognition microphone array collects and preprocesses the speech signal to remove noise interference. The preprocessed speech signal is then transmitted to the speech recognition module, which converts it into text-based commands.

[0041] Command understanding and decision-making: The CPU inputs text commands into the large language model, which then interprets the commands' semantics and identifies their intent, determining whether the user's intention is to request information, request an action, or engage in casual conversation. Based on the different intents, the model generates corresponding decision-making commands, such as moving to a specified location, grabbing a specific object, or answering a question. Specifically:

[0042] Step 21: First, complete the question in the language instruction, pass the question along with the context information to the big model, and set a prompt to clearly complete the question;

[0043] Step 22: After the question is completed, continue to call the large model and set the intent classification in the prompt. Currently, the design command classification is divided into four types: data query, security anomaly recognition, camera control, and chat. Data query uses time matching and method matching. After the matching is completed, event matching is performed, and then the method is executed. After the execution is completed, it returns to the large language model. When the event matching matches an event, it directly returns to the large language model. Security anomaly recognition uses the large language model to infer, filter out the scene, and then use it as the output of the large language model. At the same time, the instructions of the large language model are rewritten as the output of the large language model. Camera control is to enlarge the person and rotate it to the left. After the person is enlarged, omdet (real-time target detection large model) is used to give a box according to the target object, and finally send the command to isup (device registration protocol, mainly used for the device to register with the platform and service. This protocol can be used to perform command control and voice intercom with the device). After the large model understands the semantics of the question, it uses the Softmax function to perform intent classification probability distribution and select the intent with the highest score. The Softmax function formula is as follows:

[0044]

[0045] Among them, zk is the score of the kth intent output by the model.

[0046] Instruction execution: After understanding the intent, the highest-scoring intent is selected and continues to run according to the corresponding internal instructions, or calls the large model, calls the system custom function, or calls the camera instruction to control the device to complete the implementation of the user's instruction.

[0047] Action Execution and Feedback: The central processing unit (CPU) controls the movements of the mobile platform and robotic arm based on decision-making instructions, enabling physical interactions such as autonomous navigation, obstacle avoidance, grasping, and manipulation. During action execution, the sensor system monitors environmental changes and action execution in real time, transmitting feedback to the CPU. The CPU adjusts action parameters based on this feedback to ensure accuracy and safety. Simultaneously, the CPU provides user feedback on the completion of the action in the form of voice or text.

[0048] Multi-turn dialogue and context understanding: The large language model is capable of multi-turn dialogue. During interaction with the user, it can understand the user's subsequent instructions based on contextual information. For example, if the user first says, "Please go to the living room," the large language model will interpret the movement context within the instruction as "go to the living room." After the embodied agent moves to the living room, the user then says, "Help me get the remote control from the sofa." This instruction does not contain further movement context instructions, such as "then go to the master bedroom." Therefore, the model understands that this is an operation instruction in the living room context and controls the robotic arm to complete the action of picking up the remote control. Through multi-turn dialogue and context understanding, the naturalness and coherence of human-computer interaction are improved.

[0049] The embodied intelligent body structure of this embodiment is specifically constructed in the following manner: first, hardware is constructed;

[0050] Mobile platform construction:

[0051] A wheeled mobile platform is used, equipped with four high-precision motor-driven wheels, each equipped with an independent driver and encoder. A lidar is installed at the front of the mobile platform to scan the environment ahead, obtaining information on obstacle distances and terrain contours. Ultrasonic sensors are installed on both sides for close-range obstacle avoidance and to assist the lidar in environmental perception. A camera is installed on the top to identify target objects and scene information.

[0052] A reasonable power management system is designed for the mobile platform, equipped with high-capacity batteries, and installed with voltage monitoring and charging management modules to ensure stable operation and long battery life of the mobile platform.

[0053] Operating the robotic arm:

[0054] A six-degree-of-freedom manipulator arm was selected, with high-precision motors and encoders installed at each joint to achieve precise posture control. A replaceable gripper actuator was installed at the end of the arm. The gripper possesses sufficient gripping force and flexibility to grasp objects of various shapes and sizes. Furthermore, a force sensor was equipped to provide feedback during gripping, preventing damage to the object due to excessive gripping force or drop due to insufficient gripping force.

[0055] Independent control circuits and drive circuits are designed for the robotic arm, which are connected to the central processing unit for high-speed communication to ensure the real-time and accuracy of the robotic arm's movements.

[0056] Sensor system construction:

[0057] Multiple voice recognition microphones are installed at appropriate locations on the mobile platform and robotic arm to form a microphone array, improving the quality of voice signal acquisition and anti-interference capabilities. At the same time, touch sensors are installed in the gripper of the robotic arm to sense tactile information about objects, such as hardness and surface texture.

[0058] Connect the data interfaces of all sensors to the central processing unit, and perform data calibration and calibration to ensure the accuracy and consistency of sensor data.

[0059] CPU setup:

[0060] A high-performance computing chip is used as the core of the central processing unit (CPU), equipped with sufficient memory and storage space to meet the operational requirements of large language models. A cooling system is designed for the CPU, using fan cooling or liquid cooling to ensure stability during long-term high-load operation.

[0061] Install the operating system and related drivers on the central processing unit, build the model deployment environment and sensor data processing framework, and prepare for subsequent software development and system integration.

[0062] Then carry out software development and system integration

[0063] Large language model integration:

[0064] Select an open-source large language model as the base model, such as a GPT model or its variants. Collect natural language data relevant to embodied agent application scenarios, including smart home commands, industrial operation instructions, and everyday conversations, and fine-tune the model. During training, adjust model hyperparameters, such as the learning rate and batch size, to optimize performance.

[0065] The trained model is converted into a format suitable for running on a central processing unit (CPU). Model quantization technology is used to quantize the model's weights from floating-point numbers to low-order integers, reducing the model's storage space and computational complexity. At the same time, hardware acceleration technologies, such as GPU acceleration, are used to accelerate the model's inference process and improve its response speed.

[0066] The model inference framework is deployed on the central processing unit (CPU), taking sensor data and user instructions as input, invoking the model for inference, and obtaining the model's output. The model's output is then post-processed, such as through decoding and format conversion, to convert it into executable decision instructions.

[0067] Human-computer dialogue process development:

[0068] Develop a speech recognition module, use deep learning algorithms to recognize the voice signals collected by the microphone array, convert the voice signals into text, call the large language model for inference, and after successful inference, convert the obtained text into voice signals or instructions and pass them to the camera for voice broadcast or control.

[0069] In summary, the embodied intelligent agent structure of this embodiment effectively addresses the lack of embodiment in existing human-computer dialogue systems, which are unable to move and operate autonomously in the physical world. Existing systems also have shortcomings in understanding and generating natural language, failing to accurately grasp user intent and generate fluent and accurate responses.

[0070] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An embodied intelligent agent structure for human-computer dialogue based on a large model, characterized by: include: An embodied structure, which is an embodied agent used to move and operate autonomously in the outside world; The large language model is deployed on the central processor within the embodied structure to understand the language commands issued by the user and drive the specific structure to move and operate autonomously based on the understood user language commands; The specific steps for the large language model to understand language instructions are as follows: In step 1, the user issues a command to the embodied structure through voice. The voice recognition microphone array collects and preprocesses the voice signal, and then transmits the preprocessed voice signal to the voice recognition module, which converts it into a text-based command. In step 2, the CPU inputs the text instructions into the large language model, which performs semantic understanding and intent recognition on the instructions. Based on different intents, the model generates corresponding decision instructions. Step three: The central processing unit drives the embodied structure to move and operate autonomously according to the decision instructions.

2. The embodied intelligent agent structure for realizing human-computer dialogue based on a large model according to claim 1, characterized in that: The embodiment structure includes: The mobile platform is a wheeled or tracked structure equipped with laser radar, cameras, and ultrasonic sensors to sense the surrounding environment and obtain information about obstacles, terrain, and the location of target objects; An operating robot arm, mounted on a mobile platform, is used to grasp and manipulate objects; A sensor system, comprising a voice recognition microphone array and a touch sensor for receiving user voice commands and sensing tactile information of objects; The central processing unit is connected to the mobile platform, sensor system and operating robotic arm to receive voice commands output by the sensor system, perceive tactile information of objects, obtain obstacle information, terrain information and target object position, and control the operating robotic arm to grasp and operate objects.

3. The embodied intelligent agent structure for realizing human-computer dialogue based on a large model according to claim 1 or 2, characterized in that: The large language model is obtained by first selecting a large language model, and then further fine-tuning the large language model according to the application scenario of the embodied structure.

4. The embodied intelligent agent structure for realizing human-computer dialogue based on a large model according to claim 1 or 2, characterized in that: The central processing unit in step three drives the embodied structure to move and operate autonomously according to the decision instructions in the following specific manner: the central processing unit controls the movement of the mobile platform and the robotic arm according to the decision instructions to realize autonomous navigation, obstacle avoidance, grasping, and operation. In the process of executing the action, the sensor system monitors the environmental changes and the execution of the action in real time, and transmits the feedback information to the central processing unit. The central processing unit adjusts the action parameters according to the feedback information. At the same time, the result of the action execution is fed back to the user in the form of voice or text to inform the user of the completion status of the task.

5. The embodied intelligent agent structure for realizing human-computer dialogue based on a large model according to claim 1 or 2, characterized in that: One way in which the large language model performs semantic understanding of instructions in step 2 is: after receiving a voice instruction, it understands the moving scene instruction in the voice instruction and drives the embodied structure to move to the scene. Then, it receives the next language instruction and understands whether it contains a moving scene instruction. If so, it executes the moving scene instruction first and drives the embodied structure to move to the scene. If not, it determines that the language instruction is the current scene instruction, and controls the operation of the robotic arm to grasp or operate the current scene objects based on the language instruction.

6. The embodied intelligent agent structure for realizing human-computer dialogue based on a large model according to claim 1 or 2, characterized in that: The specific steps of the large language model in step 2 for semantic understanding of the instruction are: Step 21: First, complete the question in the language instruction, pass the question along with the context information to the big model, and set a prompt to clearly complete the question; Step 2: After completing the question, continue to call the big model and set the intent classification in the prompt. Currently, the design command classification is divided into four types: data query, security anomaly identification, camera control, and chat. After the big model understands the semantics of the question, it uses the Softmax function to perform intent classification probability distribution and selects the intent with the highest score. The Softmax function formula is as follows: Among them, zk is the score of the kth intent output by the model.

Images