Indoor barrier-free guidance and service system and method based on robot

Abstract

The invention discloses an indoor barrier-free guidance and service system and method based on a robot, which applies a robot AI and navigation map technology to an innovative project in the field of barrier-free assistance of an indoor venue and provides an integrated service solution for disabled people. And multiple functions of robot calling, motion control and intelligent feedback adapted to barrier-free scenes, intelligent scheduling, identity recognition, multi-language service and the like are realized. A barrier-free service ecological system with high availability and safety is constructed, and an intelligent service solution covering a whole scene and a whole crowd is created through multi-party combination of a mobile terminal and a robot, multi-modal interaction and a scene self-adaption technology. And during leader, the destination can be changed at any time, a new route does not need to be planned after exiting, and task storage and switching capabilities are provided.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of robot guidance and service, and particularly relates to an indoor barrier-free guidance and service system and method based on a robot. BACKGROUND

[0002] Currently, there is a relatively fixed mode in the interaction of service type robots, or in the chat mode, or in other relatively simple working modes; currently, robots have basic chatting, navigation, and cruising capabilities, but in terms of language and action interaction modes, especially barrier-free interaction, there is no friendly interaction mode for the disabled group, especially the visually impaired group, which is specifically manifested in.

[0003] In the prior art, when a visually impaired person comes to a barrier-free venue, guidance and service are needed, and the robot waits to be triggered or awakened at a fixed reception point, but the robot may also be working and not at the fixed reception point. It can be seen that the prior art lacks a convenient interaction mechanism to call the robot to provide service, which is particularly important for the visually impaired group and can change the current situation of people looking for robots.

[0004] In the prior art, the existing robot cannot adjust the walking speed in time by voice control or by the force of pulling the robot handle during the walking process of guiding a visually impaired friend to the destination A. It cannot broadcast the remaining distance to the destination, turn reminders, and descriptions of the surrounding environment, and cannot provide users (especially visually impaired people) with a relatively environmental awareness and a sense of security. It can be seen that the prior art lacks a humanized design and a friendly interaction system for the visually impaired group.

[0005] In the prior art, the existing robot can only exit and then plan a new route if the user wants to go to other places during the process of leading the way to the destination A. It can be seen that the prior art lacks an intelligent task scheduling system to provide such task saving and switching capabilities.

[0006] In the prior art, the robot cannot provide different types of services for the disabled group compared to the able-bodied. SUMMARY

[0007] The present application aims to provide an indoor barrier-free guidance and service system and method based on a robot, which addresses the deficiencies in the prior art and constructs an intelligent task planning capability and an intelligent interaction capability based on the robot itself, enabling an interaction system and an interaction method that is more suitable for barrier-free assistance scenarios.

[0008] To solve the above technical problems, the following technical solutions are adopted.

[0009] A robot-based indoor barrier-free guidance and service system includes an APP, a robot, an intelligent agent, an AI cloud service, a business interface cloud service, a robot monitoring cloud service, a toolset, and a task set.

[0010] The app is used to call the robot.

[0011] The robot is used to broadcast information, execute movement commands, and perform identity recognition. The robot includes a motion control module, a vision module, an auditory module, a speaker module, a smart handrail module, a map module, and a navigation module.

[0012] An intelligent agent, mounted on a robot, serves as the core decision-making and control unit of the system, responsible for context memory, task management, state management, and command execution orchestration. This intelligent agent includes a context memory module, a task management module, a state management module, and a command execution orchestration module.

[0013] The AI ​​cloud service is used for speech recognition, speech synthesis, language processing, integrated decision-making, image recognition, voiceprint recognition, face recognition, gait recognition, and disability recognition. The AI ​​cloud service includes modules for speech recognition, speech synthesis, natural language processing, integrated decision-making, image recognition, voiceprint recognition, face recognition, gait recognition, and disability recognition.

[0014] Business interface cloud services include services provided for apps and services provided for robots.

[0015] Robot monitoring cloud service, used to control robots.

[0016] Toolset, which encapsulates the definition of the current robot's capabilities.

[0017] A task set encapsulates the definition of the types of tasks that a robot can perform.

[0018] A robot-based indoor barrier-free guidance and service method includes...

[0019] (1) Call the robot: The user wakes up the APP with voice, finds available robots through the mobile APP, and then the user and the selected robot go to meet each other.

[0020] (2) Motion control and intelligent feedback adapted to barrier-free scenarios: The robot’s motion is controlled by voice commands and intelligent handrails, and intelligent broadcast feedback is provided by the robot based on built-in maps, vision capabilities and AI cloud capabilities.

[0021] (3) Intelligent scheduling: When a user issues an intelligent scheduling command, the AI ​​cloud service will analyze and identify the task type or command hit result. If it is a command, the intelligent agent will call the robot to execute it according to the hit result. The task type includes a leading task, which generates the corresponding route and accessibility broadcast configuration by the business interface cloud service according to the user's instructions. The intelligent agent drives the robot to lead the way according to the route and accessibility broadcast configuration and continuously updates the task information.

[0022] When the destination needs to be changed during a navigation task, the user issues a voice command to change the destination. The business interface cloud service receives the command, first determines the validity of the location. If it is valid, the current task is paused through the task management module. If it is invalid, a similar location is selected for confirmation to clarify the destination. The current task destination parameters are updated, the map tool is called again to plan the navigation route, and the UI display and navigation prompts are updated.

[0023] In the process of determining validity, the location converted from the user's voice command may, due to various undesirable circumstances, result in the voice recognition model giving the most probable result, but this result cannot accurately match any of the locations in the robot's location list. For example, the recognition result may be "Take me to the accessible restroom. 3," but "accessible restroom 3" already exists in the location list and cannot be precisely matched, even though they are locations with similar meanings. The algorithms for comparing location similarity are the Levenshtein distance algorithm and LLM intent recognition. The Levenshtein distance algorithm is used in the process of calculating similar locations, while LLM intent recognition is used in the process of converting the user's voice command into a tool call. The Levenshtein distance algorithm calculates a similarity threshold. If no location with a similarity greater than 0.7 is found after traversing all locations in the robot's location list, the location is considered invalid.

[0024] (4) Identity recognition: The robot collects audio and video information, and the intelligent agent performs ID matching based on the collected audio and video information. After matching, the identity tag is updated.

[0025] The identification process includes the following steps for visually impaired individuals: The robot's camera collects image data, which is then labeled using an annotation tool to identify features such as face, posture, cane, guide dog, and the guiding relationship between two people. An open-source visual model training method is used to train a detection model, which outputs the region and visual impairment category information. The trained model is deployed on the robot, and upon detecting the visual impairment category information in the robot's camera, it adjusts its service mode to visual impairment mode. The visually impaired person interacts with the robot multiple times, requiring the robot to remember them. The robot then calls the FaceID interface service to generate a FaceID, which, along with the corresponding photo, is stored in the robot's cloud storage.

[0026] (5) Multilingual service: The robot listens to the ambient voice, and after the AI ​​cloud service analyzes and recognizes it, it generates a voice reply in the corresponding language. The intelligent agent calls the robot to broadcast the voice reply.

[0027] The above technical solution has the following beneficial effects.

[0028] This invention is an innovative project that applies robot AI and navigation map technology to the field of accessibility assistance in indoor venues, providing an integrated service solution for people with disabilities. It constructs a highly available and secure accessibility service ecosystem, creating an intelligent service solution covering all scenarios and all population groups through multi-modal interaction and scene adaptation technologies, combining mobile terminals and robots. Attached Figure Description

[0029] The invention will now be further described with reference to the accompanying drawings.

[0030] Figure 1 This is a diagram showing the overall architecture of the guidance and service system of this invention.

[0031] Figure 2 This is a flowchart for a scenario involving calling a robot.

[0032] Figure 3 This is a flowchart of motion control based on voice commands.

[0033] Figure 4 This is a flowchart of motion control based on smart handrails.

[0034] Figure 5 This is a flowchart of the intelligent broadcasting process based on built-in maps, visual capabilities, and AI cloud capabilities.

[0035] Figure 6 This is a flowchart for an intelligent scheduling scenario.

[0036] Figure 7 This is a flowchart of an identity recognition method.

[0037] Figure 8 A flowchart for multilingual service methods.

[0038] In the attached diagram, the word "module" has been omitted from each submodule to reduce the number of characters. Detailed Implementation

[0039] This invention aims to provide a robot-based indoor barrier-free guidance and service system and method. Based on the robot's own capabilities, it constructs an interactive system and method with intelligent task planning and intelligent interaction capabilities, which is more adaptable to barrier-free assistance scenarios.

[0040] The technical solution of the present invention will be described in detail below with reference to specific embodiments. Example 1.

[0041] like Figure 1 As shown, an indoor barrier-free guidance and service system based on robots includes a user, an APP, a robot, an intelligent agent, an AI cloud service (also known as AI cloud), a business interface cloud service (also known as business cloud), a robot monitoring cloud service, a toolset, and a task set.

[0042] User: User.

[0043] Accessible travel application: APP, used to call the robot.

[0044] Robot: Used to broadcast information, execute movement commands, and perform identity recognition, it includes the following sub-modules.

[0045] 1. Motion Control Module: Robot.MOV, which connects the robot's chassis system and head control system, and is used to control the robot to execute motion commands.

[0046] 2. Vision Module: Robot.VISION, which connects to the cameras on the robot's body, chest, and back to acquire visual information.

[0047] 3. Hearing module: Robot.HEARING, which connects to the array microphones on the robot's head and is used to acquire sound information from the external environment.

[0048] 4. Speaker Module: Robot.SPEAKER, which connects to the robot's speaker and is used to convert electrical signals into sound to achieve sound output.

[0049] 5. Intelligent Handrail Module: Robot.BACKHAND, connected to the intelligent handrail on the back of the robot, is used to complete motion control based on the intelligent handrail. Users can adjust the intelligent handrail to cause changes in the position of the push and pull torque acquisition sensor. After analysis and recognition, the sensor generates a response result, thereby realizing motion control.

[0050] 6. Map module: Robot.MAP, connects to the SLAM map toolset, used for building environmental maps, localization, and path planning.

[0051] 7. Navigation Module: Robot.NAVI, connects the navigation toolset and works in conjunction with the map module for path planning, dynamic obstacle avoidance, and motion control.

[0052] An intelligent agent, which uses robots as carriers, includes the following sub-modules.

[0053] 1. Context Memory Module: Agent.MEM is used to store, associate, and recall historical interaction information and environmental states, enabling the agent to have memory capabilities. Its core role is reflected in maintaining interaction continuity, dynamic personalized adaptation, and complex task decomposition, so as to achieve more coherent and personalized interaction.

[0054] 2. Task Management Module: Agent.TASKMGR is used for task decomposition, scheduling, execution monitoring and coordination. Its core functions are reflected in task planning and decomposition, suspension and resumption, execution monitoring and fault tolerance, and multi-task collaboration, which can efficiently coordinate multiple tasks and achieve reliable completion of complex tasks.

[0055] 3. Status Management Module: Agent.STATEMGR is used to uniformly maintain, update and distribute robot status information. Its core functions are reflected in status tracking and maintenance, status transition control, multi-module status synchronization and fault tolerance and recovery, ensuring the consistency and reliability of system behavior.

[0056] 4. Command execution orchestration module: Agent.CHOREOGRAPHER is used for atomic command combination and flow control, and to aggregate command execution context information. Its core functions are reflected in command combination, flow control, fault tolerance in the execution process, and aggregation of command execution context information, solving the problems of dependency management, sequence control, and exception handling of multiple commands in complex tasks.

[0057] AI Cloud Service: AICloud, and the sub-modules involved in the feedback control process. AI Cloud Service can also be called AI Cloud, as shown in the attached diagram.

[0058] 1. Speech Recognition Module: AICloud.ASR, which integrates intelligent voice services for speech recognition and language identification.

[0059] 2. Speech Synthesis Module: AICloud.TTS, which integrates intelligent speech services for multilingual speech synthesis.

[0060] 3. Natural Language Processing Module: AICloud.NLP, which integrates natural language processing services for multilingual domain intent definition.

[0061] 4. Integrated Decision Module: AICloud.MIND comprehensively analyzes multi-source information, weighs optimization objectives, generates optimal strategies, and coordinates the execution of various sub-modules to solve uncertainties, multi-objective conflicts, and real-time response issues in complex scenarios.

[0062] 5. Image Recognition Module: AICloud.IMGRECOG, used to analyze, understand and process image information.

[0063] 6. Voiceprint recognition module: AICloud.SOUNDID, which identifies the user by analyzing voice information.

[0064] 7. Face recognition module: AICloud.FACEID, which identifies individuals by analyzing facial images or facial features.

[0065] 8. Walking posture recognition module: AICloud.POSEID, which identifies individuals by analyzing their walking posture.

[0066] 9. Disability Identification Module: AICloud.HANDICAPRECOG, which is a core component of barrier-free interaction, uses multimodal perception technology to detect, identify and assist people with mobility or sensory impairments in real time.

[0067] 10. Chat module: AICloud.CHAT, which integrates a multi-language chat service with a large model.

[0068] Business Interface Cloud Service: BuzCloud, also known as Business Cloud (as shown in the attached diagram), includes...

[0069] 1. Services provided to the APP: BuzCloud.APP, also known as Business Cloud.Application, as shown in the attached diagram.

[0070] 2. Services provided for the robot: BuzCloud.ROBOT, also known as Business Cloud.Robot, is shown in the attached diagram.

[0071] Robot monitoring cloud service: RobotCloud, also known as robot cloud, includes services for controlling robots: RobotCloud.ROBOTCTRL.

[0072] Toolset: The ToolSet encapsulates the definition of the current robot's capabilities.

[0073] TaskSet encapsulates the definition of the types of tasks that the robot can perform. Example 2.

[0074] A robot-based indoor barrier-free guidance and service method, which is based on the above-mentioned system and includes...

[0075] I. Scenarios involving calling robots.

[0076] Users discover a list of available accessible robots, along with their distance and direction, through the app. Based on the robot's status and location, the app prompts the user to select a reception point. The user selects a reception point in the app, and the robot arrives at the selected point. The app then queries the cloud to check the robot's status and displays its remaining distance to the user. The user uses the app's photo navigation function, which, after being analyzed and recognized by AI cloud services, generates a route navigation plan. Following this plan, the user navigates to the selected reception point. The robot arrives at the reception point and announces its location. The user then goes to the reception point and finds the robot by following its voice. Figure 2 As shown, the specific implementation process is as follows.

[0077] Phase 1: Users travel to the vicinity of the accessible venue by means of transportation.

[0078] 1. Users open the App and approach the robot deployment venue.

[0079] 1.1 The App prompts users to find the number of available accessibility robots and their distances.

[0080] 1.1.1 App → BuzCloud, request parameters include location coordinates.

[0081] 1.1.2, BuzCloud → App, the response parameters include the list of currently available accessibility robots as well as distance and direction.

[0082] Phase Two: The Two-Way Journey Between Users and Robots.

[0083] 1. The robot arrives at the reception point.

[0084] 1.1 The App prompts the user to select a reception point (so that the robot can come and provide services) based on the robot's status and the phone's location.

[0085] 1.1.1 App → BuzCloud, request parameters include location coordinates and robot ID.

[0086] 1.1.2, BuzCloud → App, the response parameters include a list of locations the robot can reach.

[0087] Users select a reception point in the app.

[0088] 1.2.1 App → BuzCloud, request parameters include location coordinates, robot ID, and location ID.

[0089] 1.2.2, BuzCloud → RobotCloud, request parameters include robot ID and location ID.

[0090] 1.2.3 RobotCloud → BuzCloud, the response parameters include the result of this command execution.

[0091] 1.2.4. BuzCloud → App, the response parameters include the task ID executed by the robot.

[0092] 1.3 The App queries the cloud status and presents the robot's remaining distance information to the user.

[0093] 1.3.1, App → BuzCloud, request parameters include location coordinates, robot ID, and task ID.

[0094] 1.3.2, BuzCloud → RobotCloud, the request parameters include the robot ID.

[0095] 1.3.3 RobotCloud → BuzCloud, the response parameters include the robot's status information, including the robot's location and other information.

[0096] 1.3.4. BuzCloud → App: The response parameters include the robot's task execution status, which includes the remaining distance to the designated location.

[0097] 2. The stage when the user approaches the reception point.

[0098] 2.1 Users use the App's photo navigation function.

[0099] 2.1.1 App → AICloud, request parameters include environmental photos.

[0100] 2.1.2 AICloud → App, the response parameters include a description of the current environment.

[0101] 2.1.3. When using the App to access BuzCloud, the request parameters include the current location, environment description, and reception point location.

[0102] 2.1.4. BuzCloud → AICloud: The request parameters include the current location, environment description, reception point location, map information, and target point location information.

[0103] 2.1.5 AICloud → BuzCloud, the response parameters include the path scheme.

[0104] 2.1.6 BuzCloud → App, the response parameters include the path navigation scheme.

[0105] 2.2 Users navigate to the selected reception point using the route navigation scheme.

[0106] Phase 3: The meeting between the user and the robot.

[0107] 1. After the robot arrives at the reception point, it waits in place and announces its location.

[0108] 2. After arriving near the reception point, the user finds the robot by following the sound.

[0109] II. Motion control and intelligent feedback methods adapted to barrier-free scenarios.

[0110] For voice-command-based motion control, the user issues a voice command, which is then parsed and recognized by the AI ​​cloud service to generate a command response. The intelligent agent then invokes the robot to provide prompts or perform motion control based on the response. For motion control based on smart handrails, the user issues button shortcut commands via the handrail. The AI ​​cloud service then parses and recognizes these commands to generate a command response. The intelligent agent then invokes the robot to provide a description of the situation and follow-up commands or to perform motion control based on the response. For intelligent broadcasting based on built-in maps, vision capabilities, and AI cloud capabilities, the intelligent agent queries robot information, identifies the current environment, and queries the current road segment configuration information. The AI ​​cloud service then simplifies the robot information, environment recognition results, and road segment configuration information, and invokes the robot to broadcast the simplified information. The specific implementation process is as follows.

[0111] 1. Motion control based on voice commands, such as Figure 3 As shown.

[0112] 1.1 User → AICloud.ASR, request parameters include: KuaiDian.pcm.

[0113] 1.2 AICloud.ASR → Agent, response parameters include: quick, response language and text recognition result.

[0114] 1.3 Agent → AICloud.NLP, request parameters include user command text.

[0115] 1.4 AICloud.NLP → Agent, response parameters include user commands and hit results (also known as intent, embodied in...) Figure 3 middle).

[0116] 1.5 Agent → AICloud.MIND, request parameters include user commands and hit results, robot status, environmental status information, and context information.

[0117] 1.6 AICloud.MIND → Agent, the command response includes two cases.

[0118] 1.6.1. Adhere to user wishes: motion control parameters.

[0119] 1.6.2 Other reasons (emergencies, etc.): Broadcast the current situation and prompt for follow-up commands.

[0120] 1.7 Agent → Robot: The agent returns parameters following the steps above.

[0121] 1.7.1 Robot.SPEAKER, executes prompts and broadcasts.

[0122] 1.7.2 Robot.MOV performs motion control.

[0123] 2. Motion control and feedback based on smart handrails, such as Figure 4 As shown.

[0124] 2.1 Motion control.

[0125] 2.1.1 User → Robot.BackHand, press the acceleration / deceleration button.

[0126] 2.1.2 Robot.BackHand → Agent, request parameters are acceleration control parameters.

[0127] 2.1.3 Agent → AICloud.MIND, request parameters include smart handrail command, robot status, environmental status information, and context information.

[0128] 2.1.4 AICloud.MIND → Agent, the response includes two cases.

[0129] 2.1.4.1. Follow user wishes: respond to motion control parameters.

[0130] 2.1.4.2 Other reasons (emergency situations, etc.): Description of the response and subsequent commands.

[0131] 2.1.5 Agent → Robot: The intelligent agent returns parameters and calls the execution mechanism according to the above steps.

[0132] 2.1.5.1 Robot.SPEAKER: Broadcasts a description of the situation and subsequent commands.

[0133] 2.1.5.2 Robot.MOV: Performs motion control.

[0134] 2.2. Grip and support perception.

[0135] 2.2.1 User → Robot.BackHand, hold the handrail.

[0136] 2.2.2 Robot.BackHand → Infrared sensor, start grip detection.

[0137] 2.2.3 Infrared sensor → Agent, report holding status (distance / obstruction / temperature).

[0138] 2.2.4 Agent → Robot.BackHand, the impact includes two situations.

[0139] 2.2.4.1. After confirming the grip, unlock the acceleration / deceleration buttons and announce "Grip secure, acceleration / deceleration can be operated".

[0140] 2.2.4.2. If the grip is unstable, maintain the safety lock and announce "Not holding firmly, please hold the handrail".

[0141] 2.3 Feedback.

[0142] 2.3.1. Agent → Robot.MOV, execute the redirection (left / right).

[0143] 2.3.2 Robot.MOV→Agent, report path changes (left / right turns).

[0144] 2.3.3, Turn left.

[0145] 2.3.3.1 Agent → Robot.BackHand.Left, triggers slight vibration (intensity / duration / rhythm).

[0146] 2.3.3.2 Agent→Robot.SPEAKER, announce "Turn left, be aware of route change".

[0147] 2.3.4. Turn right.

[0148] 2.3.4.1 Agent→Robot.BackHand.Right, triggers slight vibration (intensity / duration / rhythm).

[0149] 2.3.4.2 Agent→Robot.SPEAKER, broadcast "Turn right, be aware of route change".

[0150] 3. Intelligent broadcasting based on built-in maps, visual capabilities, and AI cloud capabilities, such as... Figure 5 As shown.

[0151] 3.1 Agent → Robot: The intelligent agent intermittently queries machine information.

[0152] 3.1.1 Robot.MOV → Agent, responds to robot location information.

[0153] 3.1.2 Robot.VISION → Agent, responds to frames in the video stream at that moment.

[0154] 3.1.3 Robot.MAP → Agent, responds with point of interest information.

[0155] 3.2 Agent → AICloud, the intelligent agent requests cloud services.

[0156] 3.2.1 AICloud.IMGRECOG → Agent, responds to the environment recognition results.

[0157] 3.3 Agent → BuzCloud.ROBOT, the agent requests the current road segment configuration.

[0158] 3.3.1 BuzCloud.ROBOT → Agent, the business cloud interface response road segment configuration information.

[0159] 3.4 Agent → AICloud.MIND, the intelligent agent requests content synthesis from the intelligent cloud.

[0160] 3.4.1 Parameters include.

[0161] 3.4.1.1 Calculate the distance between the robot's current position and the target point on the map.

[0162] 3.4.1.2 Business segment configuration.

[0163] 3.4.1.3 Environmental identification results.

[0164] 3.4.2 AICloud.MIND → Agent, provides concise broadcast information.

[0165] 3.5 Agent → Robot: The intelligent agent invokes the robot's capabilities.

[0166] III. Intelligent Scheduling Scenarios.

[0167] The agent initializes by querying the robot's basic capabilities and integrating its toolset; it also queries the business definition and integrates its state and task set; the agent registers its toolset and state / task set with the AI ​​cloud service; intelligent scheduling occurs when the user issues an intelligent scheduling command, which is parsed and identified by the AI ​​cloud service, responding with the task type or command hit result. If it's a command, the agent invokes the robot to execute it based on the hit result; if it's a task, it's categorized into Lead, Chat, Ciceroni, and Arbitrate tasks; for example... Figure 6As shown, the specific implementation process is as follows.

[0168] 1. Agent initialization.

[0169] 1.1 The intelligent agent queries the robot's various basic capabilities and integrates them into a toolset including movement, speaker, microphone, location, map, and radar.

[0170] 1.1.1 Agent → Robot: The intelligent agent queries the robot's capabilities.

[0171] 1.1.2 Robot → Agent: The parameters returned by the robot include the calling methods for devices such as movement, speakers, microphones, location, maps, and radar.

[0172] 1.2. Define the intelligent agent query business and integrate it into a set of status tasks such as welcoming guests, guiding, leading the way, chatting, and arbitration.

[0173] 1.2.1 Agent → BuzCloud, Intelligent Agent Query Business Definition.

[0174] 1.2.2, BuzCloud → Agent, Business Interface Cloud Service Returns Welcome Guide Lead Casual Chat Arbitration Definition.

[0175] 1.3 The intelligent agent registers in the cloud.

[0176] 1.3.1 Agent → AICloud: Intelligently registers its basic capability set and status task set with the AI ​​cloud service.

[0177] 1.3.2 AICloud → Agent: The AI ​​cloud service returns a successful connection to the agent.

[0178] 2. Intelligent scheduling process.

[0179] 2.1 User → AICloud.ASR, request parameters include: take me around.pcm.

[0180] 2.2 AICloud.ASR → Agent, response parameters include: take me around, response language and text recognition result.

[0181] 2.3 Agent → AICloud.NLP, request parameters include user command text.

[0182] 2.4 AICloud.NLP → Agent, the response parameters include user commands and hit results (also known as intent, embodied in...) Figure 6 ).

[0183] 2.5 Agent → AICloud.MIND, request parameters include user commands and hit results, robot status, environmental status information, and context information.

[0184] 2.6 AICloud.NLP → Agent: The response parameters include the task type or the hit result of the command. If it is a task, it is divided into Ciceroni, Chat, Lead, and Arbitrate types.

[0185] 2.6.1, Lead.

[0186] 2.6.1.1 Agent → BuzCloud.Lead, the request parameters contain the necessary target parameters of the Lead task, in this case it is Zhuanzhuan.

[0187] 2.6.1.2, BuzCloud → Agent, the response parameters include the corresponding route and accessibility broadcast configuration.

[0188] 2.6.1.3 Agent → User: Drive the robot to follow the route and broadcast configuration.

[0189] 2.6.1.4 Agent → Agent.MEM, the request parameters contain continuously updated information.

[0190] When a Lead task requires a change of destination, the user issues a voice command to change the destination. The business interface cloud service receives the command, first determines the validity of the location. If valid, the task is paused through the task management module. If invalid, a similar location is selected for confirmation to clarify the destination. The current task destination parameters are updated, the map tool is called again to plan the navigation route, and the UI display and navigation prompts are updated.

[0191] In the process of determining validity, the location converted from the user's voice command may, due to various undesirable circumstances, result in the voice recognition model giving the most probable result, but this result cannot accurately match any of the locations in the robot's location list. For example, the recognition result may be "Take me to the accessible restroom. 3," but "accessible restroom 3" already exists in the location list and cannot be precisely matched, even though they are locations with similar meanings. The algorithms for comparing location similarity are the Levenshtein distance algorithm and LLM intent recognition. The Levenshtein distance algorithm is used in the process of calculating similar locations, while LLM intent recognition is used in the process of converting the user's voice command into a tool call. The Levenshtein distance algorithm calculates a similarity threshold. If no location with a similarity greater than 0.7 is found after traversing all locations in the robot's location list, the location is considered invalid.

[0192] 2.6.2, Chat.

[0193] 2.6.2.1 Agent → AICloud.Chat, the request parameters include the user's voice-to-text result, i.e., requesting casual conversation.

[0194] 2.6.2.2 AiCloud.Chat → Agent, the response parameters include context-based dialogue results, i.e., casual conversation replies.

[0195] 2.6.2.3 Agent → User: The message is presented to the user in the form of text display on screen and voice broadcast.

[0196] 2.6.2.4 Agent → Agent.MEM, the request parameters contain continuously updated information.

[0197] 2.6.3, Ciceroni.

[0198] 2.6.3.1 Agent → BuzCloud.Ciceroni, the request parameters include the user's speech-to-text results, i.e., requesting a navigation.

[0199] 2.6.3.2, BuzCloud.Ciceroni → AICloud.plan, the request parameters include map information, location configuration information, and accessibility broadcast configuration.

[0200] 2.6.3.3, AICloud.plan → BuzCloud.Ciceroni, the response parameters include the route planning scheme and the description of the points of interest.

[0201] 2.6.3.4, BuzCloud.Ciceroni → Agent, the response parameters include this navigation scheme.

[0202] 2.6.3.5 Agent → User, start the guided tour with the robot.

[0203] 2.6.3.6 Agent → Agent.MEM, the request parameters contain continuously updated information.

[0204] 2.6.4 Arbitrate.

[0205] 2.6.4.1 Agent → Agent.TASKMGR, the request parameters contain the user command that needs to be arbitrated.

[0206] 2.6.4.1.1, Agent.TASKMGR → Agent.MEM, request parameters include start and end, time period, and memory filtering conditions.

[0207] 2.6.4.1.2, Agent.TASKMGR → Agent.STATEMGR, requests current status information.

[0208] 2.6.4.1.3, Agent.TASKMGR → Agent.CHOREOGRAPHER, requests information on the current command execution.

[0209] 2.6.4.2 Agent.TASKMGR performs rule-based control and ultimately outputs the following task control behaviors.

[0210] 2.6.4.2.1. Original task paused - new task started.

[0211] 2.6.4.2.2, Continue the original task - add a new task.

[0212] 2.6.4.2.3 Continue the original task - cancel the new task.

[0213] 2.6.4.2.4 Original task canceled - new task started.

[0214] IV. Identity Verification.

[0215] The user wakes up the robot, and the intelligent agent activates the robot's array microphone and camera to collect raw audio from the sound source, estimated spatial location of the sound source, raw video from the camera, and spatial pitch angle information. The AI ​​cloud service establishes correspondences between voiceprint IDs and time-domain / frequency-domain data, between face IDs and image regions, between gait posture IDs and image regions, and between individuals with disabilities and image regions. The intelligent agent integrates the spatial location of the sound source, the spatial pitch angle parameters of the head camera, and the correspondences established by the AI ​​cloud service to perform ID pairing. After pairing, it checks if a pair exists. If one exists, its confidence level is increased and its identity tag is updated; otherwise, its confidence level is initialized to 0 and its identity tag is updated. The specific implementation process is as follows.

[0216] 1. User → Agent: The user wakes up the robot.

[0217] 2. Agent → Robot: The intelligent agent requests data from the robot's array microphone and camera.

[0218] 2.1 Robot.HEARING → Agent, responds to the original audio of the spatial sound source and its estimated location information.

[0219] 2.2 Robot.VISION → Agent, responds to the camera's raw video and the pitch angle information of the shooting space.

[0220] 3. Agent → AICloud: The intelligent agent requests AI cloud services, with parameters including raw audio and video.

[0221] 3.1 AICloud.SOUNDID → Agent: The correspondence between the voiceprint ID and the time-domain and frequency-domain of the response agent after audio calculation.

[0222] 3.2 AICloud.FACEID → Agent: The correspondence between the face ID and the image region after the intelligent agent performs video calculations.

[0223] 3.3 AICloud.POSEID → Agent: The correspondence between the walking posture ID and the image area after the intelligent agent's video calculation.

[0224] 3.4 AICloud.HANDICAPRECOG → Agent, responds to the agent with the calculated correspondence between the person with the disability and the screen area.

[0225] 4. The Agent performs ID matching based on the spatial location of the sound source, the spatial pitch angle parameters of the head camera, and several sets of corresponding relationships returned by AICloud. After matching, the following steps are performed.

[0226] 4.1. Agent → Agent.MEM, query whether a pair already exists.

[0227] 4.1.1 If it already exists, increase its confidence level and update the identity tag.

[0228] 4.1.2 If it does not exist, initialize its confidence level to 0 and update the identity tag.

[0229] The process of identifying visually impaired individuals: The robot's camera collects image data, and the image is labeled using a labeling tool to identify the following features: face, posture, white cane, guide dog, and the guiding relationship between two people. An open-source visual model training method is used to train a detection model, and the model outputs the region and visual impairment category information. The trained model is deployed on the robot, and after the robot's camera detects the visual impairment category information, it adjusts its service mode to visual impairment mode. The visually impaired person interacts with the robot multiple times, requiring the robot to remember them. The robot calls the FaceID interface service to generate a FaceID, and the generated FaceID and the corresponding photo are stored in the robot's cloud storage.

[0230] Multilingual service methods.

[0231] The robot listens to ambient speech, which is then analyzed and recognized by the AI ​​cloud service to generate a corresponding voice response. The intelligent agent then invokes the robot to read this voice response aloud, such as... Figure 8 As shown, the specific implementation process is as follows.

[0232] 1. Robot.HEARING → Agent: The robot listens to ambient sounds.

[0233] 2. Agent → AICloud.ASR: The agent requests AI cloud services, and the parameters include the original audio.

[0234] 3. AICloud.ASR → Agent, responds with language and text recognition results.

[0235] 4. Agent → AICloud.CHAT, request chat service.

[0236] 5. AICloud.CHAT → Agent, responding to chat results.

[0237] 6. Agent → AICloud.TTS: The intelligent agent requests AI cloud services, including language and text responses, and casual conversation replies.

[0238] 7. AICloud.TTS → Agent, generate voice responses in the corresponding language.

[0239] 8. Agent → Robot.SPEAKER, the intelligent agent calls upon the robot's capabilities to broadcast.

[0240] The above are merely specific embodiments of the present invention, but the technical features of the present invention are not limited thereto. Any simple changes, equivalent substitutions, or modifications made based on the present invention to solve essentially the same technical problems and achieve essentially the same technical effects are all covered within the protection scope of the present invention.

Claims

1. A robot-based indoor barrier-free guidance and service system, characterized by comprising: The APP, the robot, the intelligent agent, the AI cloud service, the business interface cloud service, and the robot monitoring cloud service; The APP is used for calling the robot; The robot is used for completing information broadcasting, executing motion instructions, and identity recognition; The intelligent agent is loaded on the robot and serves as a core decision and control unit of the system, responsible for context memory, task management, state management, and command execution arrangement; The AI cloud service is used for voice recognition, voice synthesis, language processing, integrated decision, image recognition, voiceprint recognition, face recognition, gait recognition, and disability recognition; The business interface cloud service includes services provided for the APP and services provided for the robot; The robot monitoring cloud service is used for controlling the services of the robot. 2.The robot-based indoor barrier-free guiding and serving system according to claim 1, characterized in that: The robot includes a motion control module, a vision module, an auditory module, a speaker module, an intelligent handrail module, a map module, and a navigation module; the motion control module is connected with a body chassis system and a head control system of the robot; the vision module is connected with a camera of the robot; the auditory module is connected with an array microphone of the robot; the speaker module is connected with a speaker of the robot; the intelligent handrail module is connected with an intelligent handrail of the robot; and the map module is connected with a SLAM map tool set; The navigation module is connected with a navigation tool set. 3.The robot-based indoor barrier-free guiding and serving system according to claim 1, wherein: The intelligent agent includes a context memory module, a task management module, a state management module, and a command execution arrangement module; The context memory module is used for storing, associating, and calling historical interaction information and environment states; The task management module is used for task decomposition, scheduling, execution monitoring, and coordination; The state management module is used for unified maintenance, update, and distribution of state information of the robot; The command execution arrangement module is used for combination of atomic commands and flow control aggregation of command execution context information. 4.The robot-based indoor barrier-free guiding and serving system according to claim 1, wherein: The AI cloud service includes a voice recognition module, a voice synthesis module, a natural language processing module, an integrated decision module, an image recognition module, a voiceprint recognition module, a face recognition module, a gait recognition module, and a disability recognition module.

5. A robot-based indoor barrier-free guidance and service method, characterized by The robot includes: (1) Motion control and intelligent feedback suitable for barrier-free scenarios: motion control of the robot based on voice instructions and intelligent handrails, and intelligent broadcasting feedback completed by the robot based on built-in maps, vision capabilities, and AI cloud capabilities; (2) Intelligent scheduling: the user issues an intelligent scheduling instruction, and the AI cloud service analyzes and identifies the hit result of the task type or command, and if it is a command, the intelligent agent calls the robot for execution according to the hit result; The task type includes a guiding task, which generates a corresponding route and barrier-free broadcast configuration according to user instructions by the business interface cloud service, and the agent drives the robot to guide according to the route and barrier-free broadcast configuration, and continuously updates the task information; when the destination needs to be changed in the guiding task, the user issues a voice instruction to change the destination, the business interface cloud service receives the instruction, first judges the validity of the place, if valid, suspends the current executing task through the task management module, if invalid, takes a similar place to confirm, and determines the destination; update the current task destination parameter, re-call the map tool to plan the navigation route, and update the UI display and broadcast navigation prompt; (3) Identity recognition: the robot collects audio information and video information, and the agent performs ID matching according to the collected audio information and video information, and updates the identity label after matching. 6.The robot-based indoor barrier-free guiding and serving method according to claim 5, wherein: The guidance and service also include calling robots, and the user calls the APP to find available robots through the mobile phone APP, and then the user and the selected robot meet at the selected meeting point, specifically: a. The user discovers the list of available barrier-free robots, distance and direction through the APP; b. The robot comes to the reception point: the APP prompts the user to select the reception point according to the robot state and positioning location, the user selects the reception point in the APP, the robot comes to the selected reception point, and the APP presents the remaining distance information of the robot to the user by querying the cloud state; c. The user approaches the reception point: the user uses the APP to take a photo to identify the path, and generates a path navigation scheme after AI cloud service analysis and identification, and navigates to the selected reception point according to the path navigation scheme; d. The user and the robot meet: the robot arrives at the reception point and broadcasts the location, and the user arrives at the reception point and finds the robot by sound. 7.The robot-based indoor barrier-free guiding and serving method according to claim 5, wherein: The (2) motion control and intelligent feedback suitable for barrier-free scenes includes: a. Motion control based on voice instructions: the user issues a voice instruction, which is analyzed and identified by the AI cloud service to generate a command response result, and the agent calls the robot to execute the prompt broadcast or execute the motion control according to the command response result; b. Motion control and feedback based on intelligent handrails: ① Motion control: the user issues a key shortcut instruction through the intelligent handrail, and the agent calls the robot to broadcast the situation description and follow-up command or execute the motion control according to the command content; ② Holding perception: the user holds the intelligent handrail, and the agent detects the holding state to call the robot to broadcast whether the holding is stable, and locks or unlocks the motion control according to the holding state; ③ Feedback: execute the steering, trigger the slight vibration, and broadcast the steering information; c. Intelligent broadcast based on built-in map, vision capability and AI cloud capability: the agent queries the robot information, identifies the current environment and queries the current road segment configuration information, and the AI cloud service simplifies the robot information, environment identification result and road segment configuration information, and calls the robot to broadcast the simplified information. 8.The robot-based indoor barrier-free guiding and serving method according to claim 5, wherein: In the step (3) intelligent scheduling, the task type also includes a casual conversation task, a tour task and an arbitration task: Idle chat task, according to the user instruction, the AI cloud service generates the answering result of the context, the agent calls the robot to present to the user in the form of text screen display and voice broadcast, if the tool is called, the task arbitration is carried out; Guided tour task, according to the user instruction, the AI cloud service and the business interface cloud service generate the path planning scheme and the point of interest description, according to the path planning scheme and the point of interest description, the guided tour scheme is formulated, the agent drives the robot to start the guided tour explanation, and the task information is updated constantly; Arbitration task, according to the user instruction needing arbitration, the agent queries the memory, queries the current state information and queries the current command execution information, based on the queried information, the information is regularized, and finally the following task control behaviors are output: original task pause-new task start, original task continue-new task enter, original task continue-new task cancel and original task cancel-new task start. 9.The robot-based indoor barrier-free guiding and serving method according to claim 5, wherein: The step (4) identity recognition comprises: the user wakes up the robot, the agent starts the robot array microphone and camera, collects the sound source original audio, spatial position estimation, camera original video and spatial pitch angle information; the AI cloud service establishes the corresponding relationship between the voiceprint ID and the time domain frequency domain, the corresponding relationship between the face ID and the picture area, the corresponding relationship between the walking posture ID and the picture area, and the corresponding relationship between the obstacle person and the picture area, the agent integrates the sound source spatial position, the head camera spatial pitch angle parameter and the corresponding relationship established by the AI cloud service, and performs ID pairing; after pairing, it is inquired whether there is pairing, if there is, the confidence degree is increased, and the identity mark is updated, if there is not, the confidence degree is initialized to 0, and the identity mark is updated. 10.The robot-based indoor barrier-free guiding and serving method according to claim 9, wherein: The identity recognition comprises blind person identification: the robot camera collects image data, obtains image data, uses a labeling tool to label the image, and identifies the following features: face, posture, blind stick, guide dog and leading relationship of two people passing through, uses an open source visual model training method to train a detection model, and the model outputs area and category information; the trained model is deployed on the robot, and after the robot camera detects the blind category information, the service mode of the robot is adjusted to the blind mode; the blind person and the robot have multiple rounds of interaction, and the robot is required to remember itself; the robot calls the FaceID interface service to generate FaceID, and the generated FaceID and the corresponding photo at the moment are stored in the robot cloud storage.

Images