A dual-mode operation control method and system for mobile robots

Abstract

This invention discloses a dual-mode operation control method and system for a mobile robot, relating to the field of human-computer interaction technology. The method includes: acquiring the spatial pose of a remote terminal; if it exceeds a boundary, triggering redirection; calculating the rotation direction and distance of the remote terminal until the remote terminal faces the center of the walkable area; obtaining the rotation error at the current moment based on the rotation increments of the mobile robot and the remote terminal at the current and previous moments; obtaining the position error at the current moment based on the movement distance increment; and obtaining the angular velocity and linear velocity of the mobile robot based on the rotation error and position error, so as to control the movement of the mobile robot according to the movement of the remote terminal; in local mode, recognizing skeletal joints of the service object image based on the task of the remote terminal, calculating the position of the service object in the camera coordinate system, calculating the linear velocity and angular velocity of the mobile robot based on the position change of the service object, and controlling the mobile robot to follow the movement of the service object to adapt to the execution requirements of different tasks.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a dual-mode operation control method and system for a mobile robot. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Currently, with the development of intelligent technologies, more and more intelligent mobile robot systems are entering practical applications, such as in hospital ward rounds and remote consultations, where intelligent mobile robots and remote control technologies are being widely studied. However, existing mobile robot systems often suffer from the following problems:

[0004] 1. Limited functionality: Existing mobile robot systems typically only support a single function, such as remote video or local navigation, lacking the flexibility to switch between different operating modes.

[0005] 2. Interaction limitations: The existing system has weak interaction capabilities and cannot support collaborative work with collaborators in multiple task scenarios at the same time.

[0006] 3. Poor User Experience: Existing mobile robot systems often face issues such as fragmented workflows and response delays during task assignment and user execution, resulting in a poor user experience. For example, in medical settings, especially for the elderly and those with mobility issues, after a doctor remotely assigns a task, users often need to frequently travel to different departments to complete various examinations and procedures using the system's remote mode. This process is cumbersome and provides a poor experience. In reality, tasks such as cognitive function screening, intervention training, basic vital sign measurement, and simple sensory response testing can be completed locally. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a dual-mode operation control method and system for mobile robots, integrating local and remote control mechanisms to adapt to the execution requirements of different tasks.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a dual-mode operation control method for a mobile robot, applied to the interaction between a remote end, a mobile robot end, and a service object end, comprising:

[0010] In remote mode, the spatial pose data of the user sent by the remote end is obtained. Based on the distance between the user and the set boundary of the walkable area, it is determined whether the user has exceeded the boundary. If the user has exceeded the boundary, redirection is triggered, and the user's rotation direction and distance are calculated until the user is facing the center of the walkable area.

[0011] The rotation error is obtained based on the rotation increment of the mobile robot and the user at the current moment, the position error is obtained based on the movement distance increment, and the angular velocity and linear velocity of the mobile robot are obtained based on the rotation error and the position error, so as to control the movement of the mobile robot according to the user's movement.

[0012] In local mode, based on the task received from the remote end, the robot identifies the skeletal joints of the acquired service object image to calculate the position of the service object in the camera coordinate system. Based on the position change of the service object, the robot calculates the linear velocity and angular velocity of the mobile robot to control the mobile robot to follow the service object.

[0013] As an alternative implementation, after triggering the redirection, the movement and viewpoint of the mobile robot are frozen. Based on the user's current orientation, the user's current position coordinates, and the center coordinates of the walkable area, the angle between the direction vector of the user's current orientation and the direction vector of the user's orientation toward the center point of the walkable area is obtained and displayed in text form on the remote end to guide the user to rotate and walk toward the center of the walkable area. At the same time, the angle is compared with a preset threshold. When the angle is less than the preset threshold, it is determined that the user is heading toward the center point of the walkable area.

[0014] As an alternative implementation, the rotation increment is obtained based on the difference between the rotation angle at the current moment and the previous moment at the mobile robot end. The rotation increment is obtained based on the difference between the user's rotation angle at the current moment and at the previous moment. The rotation error at the current moment for: ; The rotation error at time t-1;

[0015] The incremental distance is calculated based on the difference between the distance traveled by the mobile robot at the current moment and the distance traveled at the previous moment. The increment of the travel distance is obtained based on the difference between the user's travel distance at the current time and the previous time. Position error at the current moment for: ; The position error at time t-1.

[0016] As an alternative implementation method, the angular velocity of the mobile robot at the current moment and linear velocity They are respectively:

[0017] ;

[0018] ;

[0019] in, It is proportional gain. It is integral gain. It is the differential gain; for Rotation error at any given time; for Positional error at any given time.

[0020] As an alternative implementation, after identifying skeletal joints, a set of effective joints is extracted, and the pixel coordinates of the geometric center of the effective joints are used as the coordinates of the service object in the pixel coordinate system. Based on the depth information corresponding to the pixel coordinates in the acquired depth image, the coordinates of the service object in the pixel coordinate system are mapped to the coordinates in the camera coordinate system of the mobile robot.

[0021] As an alternative implementation, the process of calculating the linear velocity and angular velocity of the mobile robot based on the position change of the service object includes:

[0022] Depend on target location at any time Current location of the mobile robot Distance between As control Time linear velocity Distance error Then, in the camera coordinate system of the mobile robot, the current position of the mobile robot is... Setting it to 0 means that the mobile robot maintains a set distance from the service recipient during the following process. Select a distance between the mobile robot and the service recipient. The point is taken as the target position of the mobile robot, and thus... ;in, The X-axis and Y-axis coordinates of the service object in the camera coordinate system;

[0023] Depend on target angle at any time Current angle of the mobile robot angular error right angular velocity at time To perform control, the current angle in the camera coordinate system of the mobile robot is... It is always 0, so Furthermore, the target angle is the angle difference between the target position and the orientation of the mobile robot. Therefore, based on the position coordinates of the service object in the camera coordinate system, we can obtain... : ;in, It is the arctangent function. The X-axis and Z-axis coordinates of the target service object in the camera coordinate system;

[0024] linear velocity at time t With angular velocity They are respectively:

[0025] ;

[0026] ;

[0027] in, It is proportional gain. It is integral gain. It is the differential gain; for angular error at any given time; for Distance error at any given time; The angle error at time t-1.

[0028] Secondly, the present invention provides a dual-mode operation control system for a mobile robot, comprising:

[0029] The redirection module is configured to acquire the user's spatial pose data sent by the remote end in remote mode, determine whether it exceeds the boundary based on the distance between the user and the set walkable area boundary, and if it exceeds the boundary, trigger redirection, calculate the user's rotation direction and distance, until the user faces the center of the walkable area.

[0030] The remote control module is configured to obtain the rotation error based on the rotation increment of the mobile robot and the user at the current moment, obtain the position error based on the movement distance increment, and obtain the angular velocity and linear velocity of the mobile robot based on the rotation error and the position error, so as to control the movement of the mobile robot according to the user's movement.

[0031] The local control module is configured to, in local mode, identify skeletal joints in the acquired image of the service object based on the task received from the remote end, thereby calculating the position of the service object in the camera coordinate system, and calculating the linear and angular velocities of the mobile robot based on the position changes of the service object, so as to control the mobile robot to follow the service object.

[0032] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0033] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0034] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0036] This invention proposes a dual-mode operation control method and system for mobile robots, supporting both local and remote operation. In remote mode, the system acquires the user's spatial pose data sent from the remote end to determine whether to trigger redirection, ensuring the user faces the center of the walkable area and avoids going out of bounds. Then, the angular velocity and linear velocity of the mobile robot are obtained based on rotation and position errors, thereby controlling the robot's movement according to the user's motion. In local mode, the position of the service object in the camera coordinate system is calculated based on the task received from the remote end, and the linear and angular velocities of the mobile robot are calculated based on the position changes of the service object to control the robot to follow the service object. Combined with an interactive mobile robot, it achieves local assistance and remote guidance for the service object's task. The integration of local and remote control mechanisms adapts to the execution requirements of different tasks. It not only significantly improves the execution efficiency of remote collaborative tasks but also plays an important role in the work efficiency and interactive convenience of local collaboration. On the one hand, intelligent scheduling and efficient communication mechanisms improve the response speed and accuracy of remote services; on the other hand, it also enhances the interactive efficiency of local collaboration. For example, users can also use this system locally to more conveniently obtain relevant task data and exchange information remotely, thereby improving the efficiency and quality of collaboration.

[0037] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 This is a flowchart of the dual-mode operation control method for a mobile robot provided in Embodiment 1 of the present invention;

[0040] Figure 2 This is a diagram of the remote ward round system architecture based on a mobile robot terminal and a VR remote terminal provided in Embodiment 1 of the present invention;

[0041] Figure 3 This is a schematic diagram of the interaction between the mobile robot terminal and the VR remote terminal provided in Embodiment 1 of the present invention;

[0042] Figure 4 This is a schematic diagram of pose tracking provided in Embodiment 1 of the present invention;

[0043] Figure 5 This is a schematic diagram of redirection provided in Embodiment 1 of the present invention;

[0044] Figure 6 This is a flowchart of the autonomous following process of a mobile robot provided in Embodiment 1 of the present invention;

[0045] Figure 7 This is a schematic diagram of mobile robot following control provided in Embodiment 1 of the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0047] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0048] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0049] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0050] Example 1

[0051] This embodiment provides a dual-mode operation control method for mobile robots. It supports both local and remote operation modes, allowing for flexible switching between them. Combined with an interactive mobile robot, it enables local assistance and remote guidance for tasks performed on service recipients, making it suitable for various application scenarios, such as healthcare.

[0052] like Figure 1 As shown, it specifically includes:

[0053] In remote mode, the spatial pose data of the user sent by the remote end is obtained. Based on the distance between the user and the set boundary of the walkable area, it is determined whether the user has exceeded the boundary. If the user has exceeded the boundary, redirection is triggered, and the user's rotation direction and distance are calculated until the user is facing the center of the walkable area.

[0054] The rotation error is obtained based on the rotation increment of the mobile robot and the user at the current moment, the position error is obtained based on the movement distance increment, and the angular velocity and linear velocity of the mobile robot are obtained based on the rotation error and the position error, so as to control the movement of the mobile robot according to the user's movement.

[0055] In local mode, based on the task received from the remote end, the robot identifies the skeletal joints of the acquired service object image to calculate the position of the service object in the camera coordinate system. Based on the position change of the service object, the robot calculates the linear velocity and angular velocity of the mobile robot to control the mobile robot to follow the service object.

[0056] In this embodiment, the above control method is applied to the interaction between the remote end, the mobile robot end, and the service object end, and is uniformly scheduled by the server. The mobile robot is controlled by the mobile robot platform, the remote end is controlled by the remote control platform, and the task to be executed is transmitted from the remote end to the service object by the task interaction platform. Thus, the service object executes the task locally through the mobile robot and provides real-time feedback on the execution status.

[0057] Taking the medical scenario as an example, various medical monitoring devices can be used to monitor the physiological state and cognitive level of the service recipients; the monitoring data can be sent to the server and saved, and then displayed in virtual reality devices; medical consultation videos can also be recorded by shooting with an RGB camera, or the audio of the consultation can be converted into text records using speech-to-text and saved on the server.

[0058] The mobile robot has autonomous navigation, task execution, and information display capabilities. It can guide service recipients to perform tasks locally, such as health assessments, rehabilitation training, and functional tests. It is also equipped with a dual-mode control switching module, which allows service recipients to interact with the mobile robot via touch when they are on the same end, or to interact remotely using virtual reality devices when they are on opposite ends, thus enabling the switching of control modes.

[0059] Remote control platforms are used to provide an immersive remote operation experience, enabling remote endpoints to monitor the performance of tasks by service recipients in real time and provide guidance and feedback.

[0060] In this embodiment, the mobile robot platform sends control commands to the mobile robot through a communication interface based on the TCP protocol.

[0061] After receiving control commands, the mobile robot moves within the scene while simultaneously using LiDAR to scan objects in the scene. It then constructs a static map using LiDAR SLAM (Simultaneous Localization and Mapping) technology. Once the static map is generated, environmental points are marked on it to construct environmental semantic information. This provides positioning information for the mobile robot platform's navigation, associating the current location coordinates with the service object information, enabling the mobile robot platform to perceive the environment and patient information based on its current location.

[0062] The mobile robot platform obtains the mobile robot's current position coordinates and current navigation behavior status (e.g., idle, navigating, failed, successful) in the constructed static map. Based on the current navigation behavior status, it manages the mobile robot's current navigation task, determining whether to continue navigation. Based on the mobile robot's current position coordinates, it determines whether to move or rotate to the target location and adjusts the mobile robot's position to ensure it performs tasks such as navigation, obstacle avoidance, and position adjustment. For example, it can rotate the mobile robot based on the user's position to face the user or service object, and provide a visual display of relevant information on the touchscreen.

[0063] In addition, the motion behavior of the mobile robot is divided into multiple motion sub-tasks, the transition states between multiple motion sub-tasks are defined, and a finite state machine is constructed to manage multiple motion sub-tasks in a unified manner. Each of the multiple motion sub-tasks is regarded as a state in the finite state machine.

[0064] The system includes five motion sub-tasks: idle state, follow state, fixed-point navigation state, rotation state, and remote control state.

[0065] Simultaneously, the actions to be executed by different motion subtasks are defined in the form of callback functions, and events to trigger task transitions are defined using enumeration types. During system initialization, the functions involved in the actions are registered for each subtask, and the events and state transition conditions when switching between motion subtasks are specified. Finite state machines read events from the event queue and determine the action to be executed based on the predefined transition conditions.

[0066] In this embodiment, the mobile robot and the remote terminal (hereinafter, we take the VR (Virtual Reality) remote terminal as an example) are located in different physical spaces. The mobile robot and the VR remote terminal collect audio and video data from both ends respectively, and transmit them to the other end through the server. The VR remote terminal displays the video to the user, realizing real-time remote voice interaction between the two ends.

[0067] The system captures the user's head rotation posture information remotely via VR, and simultaneously obtains the user's position information in space; based on the user's head rotation posture information and position information in space, it combines the user's spatial pose data and sends it to the server;

[0068] Based on the user's spatial pose data and the distance to the set walkable area boundary, it determines whether the user has exceeded the boundary. If the user has exceeded the boundary, a redirection is triggered, and the user's rotation direction and distance are calculated until the user is facing the center of the walkable area. The purpose of this process is to trigger a redirection operation when the user is walking in physical space wearing VR equipment and approaches the boundary of the walkable area, adjust the direction of travel, and guide the user back to the safe range.

[0069] Then, the rotation error is obtained based on the rotation increment of the mobile robot and the user at the current moment, the position error is obtained based on the movement distance increment, and the angular velocity and linear velocity of the mobile robot are obtained based on the rotation error and the position error, so as to control the movement of the mobile robot according to the user's movement.

[0070] Secondly, in local mode, based on the task received from the remote end, the robot identifies the skeletal joints of the acquired service object image to calculate the position of the service object in the camera coordinate system. Based on the position change of the service object, the robot calculates the linear velocity and angular velocity of the mobile robot to control the mobile robot to follow the service object.

[0071] In this embodiment, taking healthcare as an example, a remote medical task interaction mechanism is constructed, enabling doctors to issue medical-related tasks to service recipients through a remote control platform. These tasks include, but are not limited to, ward rounds, consultations, health assessments, rehabilitation training, and functional testing. Service recipients complete the corresponding operations locally using a mobile robot, thereby effectively reducing reliance on traditional medical facilities and improving the timeliness and convenience of services. Furthermore, a local operation mode is also supported. The mobile robot, as the task execution carrier, can independently complete some medical examinations and tasks in a hospital or home environment, further enhancing the system's applicability and flexibility.

[0072] The following example illustrates a virtual reality-based remote ward round. Doctors enter a virtual ward through a remote control platform and use redirected walking technology to control the movement of a mobile robot for precise virtual ward rounds. Doctors can view the patient's medical records and emotional state in real time, answer questions, and provide medical advice. Simultaneously, the mobile robot can display relevant information and emotional states, and assist in completing simple health assessments.

[0073] A remote ward round system, comprised of a mobile robot and a VR remote terminal, allows medical staff to remotely operate the mobile robot for ward rounds. Doctors, wearing VR headsets, view the scene in real-time through the mobile robot's built-in RGB-D camera, adjust their head movements to control the robot's perspective and movement. Since the walkable area on the VR remote terminal is typically smaller than the navigable area on the mobile robot (e.g., within a hospital), a redirection algorithm guides the staff, ensuring they remain within the walkable area to control the robot's movement. This achieves effective motion control of the mobile robot within a limited physical space, navigating a larger environment. Data transmission is handled by a server, including spatial pose data, task status, medical consultation monitoring data, and real-time audio and video, ultimately achieving information synchronization.

[0074] To facilitate understanding of the overall solution by those skilled in the art, the following explanation and description of this embodiment are provided in conjunction with the accompanying drawings and specific implementation steps.

[0075] Figure 2 This is an architecture diagram of a remote ward round system based on a mobile robot and a VR remote terminal. The mobile robot includes the robot itself, a platform, a touchscreen, and an RGB camera; the VR remote terminal includes a remote control platform, a camera, and a VR device integrated with a smartphone; data transmission between the mobile robot and the VR remote terminal is based on a server; the server also includes an Agora real-time audio / video platform for audio / video calls between the mobile robot and the VR remote terminal.

[0076] Includes the following steps:

[0077] Step S101: The mobile robot uses an RGB camera to capture environmental audio and video data of the ward scene and transmits it to the medical staff at the VR remote end. After wearing VR equipment, the medical staff can view the environmental audio and video data captured by the mobile robot. The voice of the medical staff will also be transmitted to the mobile robot. The mobile robot and the VR remote end can conduct audio and video calls.

[0078] Figure 3 This diagram illustrates the interaction between the mobile robot and the VR remote terminal. Specifically: First, the VR remote terminal sends a task marker to the server, and then the mobile robot reads the task marker from the server; next, both the mobile robot and the VR remote terminal read the Agora SDK configuration information from the server, join the channel, and enable audio; finally, the mobile robot and the VR remote terminal conduct an audio and video call.

[0079] Step S102: Obtain the actual location and orientation of the medical staff on the VR remote terminal;

[0080] Figure 4 This is a diagram illustrating pose tracking, using a camera on a VR remote terminal to acquire the real-time position of medical staff in three-dimensional space. And use VR devices to capture the three-degree-of-freedom pose of medical staff's heads in real time. The VR remote terminal sends the position and three-degree-of-freedom pose data of the medical staff's head to the server, where it is combined into spatial pose data. This converts it into pose data for the mobile robot. , It provides data input for the remote control of mobile robots, representing the rotation angle.

[0081] in, , , The three basic angles that describe the rotation of an object in three-dimensional space correspond to rotations about the X-axis, Y-axis, and Z-axis, respectively. The pitch angle is the angle of rotation around the Y-axis. This is the yaw angle, which is the angle of rotation around the Z-axis; The roll angle is the angle of rotation around the X-axis.

[0082] Step S103: Redirection; After obtaining the spatial pose data of the medical staff, it is determined whether to redirect based on the distance between the medical staff and the boundary of the set walkable area. In this embodiment, the freeze-turn method with reset mechanism is used as the redirection algorithm.

[0083] Step S1031: First, set the walkable area for medical staff on the VR remote terminal. In this embodiment, the walkable area in the physical space is set to a rectangular area. ,like Figure 5 As shown;

[0084] ;

[0085] in, The coordinates are the positions within the walkable space. and These are the coordinates of two fixed points within the walkable area.

[0086] Step S1032: Calculate the distance to the boundary in real time based on the current spatial pose data of the medical staff. And determine whether medical staff have exceeded the boundaries:

[0087] ;

[0088] in, These are the coordinates of the VR remote user in the physical space. When If the user is outside the walkable area and their orientation is off-center from the scene, a redirect is triggered, and the user jumps to S1033; otherwise, the user jumps to S104.

[0089] Step S1033: Trigger redirection;

[0090] First, a redirection task status flag is sent to the server, freezing the movement of the mobile robot, and thus freezing the robot's viewpoint as well.

[0091] Then, based on the user's current orientation, the user's current position coordinates, and the coordinates of the center of the walkable area (i.e., the scene center coordinates), the angle between the direction vector of the user's current orientation and the direction vector of the user's orientation towards the center of the walkable area is calculated in real time. This angle information is displayed in text form on the VR device to guide the user's rotation and encourage them to walk towards the scene center. Simultaneously, the angle is compared to a preset threshold in real time. When the angle is less than the preset threshold, it is considered that the user has already moved towards the center of the walkable area, thus determining whether the user should return to the scene center.

[0092] Finally, after the adjustments are completed, a task status marker is set, and the user can continue to walk freely within the walkable area, and the mobile robot regains control.

[0093] Step S104: Network Transmission: Asynchronous communication is performed using the HTTP protocol, with the server acting as the server-side and the VR remote terminal and mobile robot terminal acting as clients. The VR remote terminal sends the captured spatial pose data and task status markers of the medical staff to the server; after receiving the request from the VR remote terminal, the server updates the current spatial pose data and task status markers; the mobile robot terminal reads the spatial pose data and task status markers from the server in real time. The VR remote terminal and the mobile robot terminal establish a connection through the server as an intermediary to perform relevant data transmission.

[0094] Step S105: Based on the changes in the spatial pose data of medical staff at the current moment and the previous moment, the user's position and head posture changes are mapped to the corresponding motion data of the mobile robot, thereby realizing remote control of the mobile robot.

[0095] Specifically:

[0096] (1) Calculate the rotation increment of the mobile robot from the previous time to the current time. :

[0097] ;

[0098] in, Let be the rotation angle of the mobile robot at time t. Let be the rotation angle of the mobile robot at time t-1.

[0099] (2) Calculate the rotation increment of the user on the VR remote terminal from the previous moment to the current moment. :

[0100] ;

[0101] in, Let be the rotation angle of the user at time t. This represents the rotation angle of the user at time t-1.

[0102] (3) After obtaining the rotation increments of the mobile robot and the user, calculate the rotation error at the current moment. :

[0103] ;

[0104] in, Let be the rotation error at time t-1.

[0105] (4) Calculate the incremental distance traveled by the mobile robot from the previous time step to the current time step. :

[0106] ;

[0107] in, This represents the Euclidean norm, which is used to calculate the magnitude of a vector. Let be the distance traveled by the mobile robot at time t; Let be the distance traveled by the mobile robot at time t-1.

[0108] (5) Calculate the incremental distance traveled by the user on the VR remote terminal from the previous time to the current time. :

[0109] ;

[0110] in, The distance the user travels at time t; This represents the distance the user traveled at time t-1.

[0111] (6) After obtaining the incremental movement distance of the mobile robot and the user, calculate the position error at time t. ,Right now:

[0112] ;

[0113] in, The position error at time t-1.

[0114] (7) Use a PID controller to calculate the angular velocity of the robot at the current moment. and linear velocity ;

[0115] ;

[0116] ;

[0117] in, It is proportional gain. It is integral gain. It is the differential gain; for Rotation error at any given time; for Positional error at any given time.

[0118] Step S106: Medical staff use virtual reality equipment to view and roam the scene, and react accordingly to the situation on site. They can change the position of the mobile robot by walking, thereby adjusting the perspective. At the same time, through the camera on the mobile robot, medical staff can communicate with patients in the ward via voice and video.

[0119] Step S107: Monitoring the physiological information of the service recipient: Capture the facial information of the service recipient using an RGB camera; locate and align the facial area of ​​the service recipient, and analyze the emotional state of the service recipient in real time using facial expression recognition technology; transmit the emotional state of the service recipient to the server; and visualize it on a mobile robot platform and virtual reality device.

[0120] Step S108: Information Visualization: The touchscreen on the mobile robot provides doctors with an intuitive display of the patient's current electronic medical records, ward round records, and other relevant information. To better facilitate collaboration between the doctor and the mobile robot, buttons are provided on the left and right sides of the interface. Touching these buttons rotates the robot 90 degrees to the left or right, respectively. Doctors can touch these buttons as needed to rotate the mobile robot to face them, making it easier for them to view the information.

[0121] Step S109: Determine whether the remote ward round has ended. If it has not ended, proceed to step S102; otherwise, end the ward round, save the recorded video and audio files to the server, and use speech-to-text technology to convert the audio during the ward round into a written record of the consultation to support the doctor's subsequent decision-making.

[0122] The above steps are a general method for remote ward rounds based on mobile robots and virtual reality technology. This allows medical staff to remotely control a mobile robot to move within the ward through natural walking, enabling precise virtual ward rounds. Medical staff can view information about the patients, including but not limited to medical records and emotional states; they can also perform medical activities tailored to the patients, including but not limited to providing medical advice and completing health assessments.

[0123] In summary, to facilitate remote ward rounds for medical staff and overcome the limitations of time and space, this embodiment proposes a remote ward round mode based on mobile robots and virtual reality, which differs from the ward round mode using interactive devices such as keyboards, gamepads, and screen display devices. Medical staff wearing VR devices can see the scene and service recipient information on the mobile robot and control the mobile robot to move in the ward through natural walking, thereby conducting immersive consultations.

[0124] In the local mode, taking health assessment and rehabilitation training as an example, a mobile robot guides the client to complete tasks based on remotely issued instructions from a doctor. These tasks include functional testing, cognitive ability assessment, limb rehabilitation training, basic vital sign monitoring, and visual and auditory response evaluation. Doctors can monitor task progress through a remote control platform and provide real-time guidance to the client. This method is suitable for the elderly, those with mobility impairments, or groups requiring long-term health management.

[0125] To address the need for screening cognitive impairment, an interactive software-based assessment program (such as cognitive training games) can be deployed on a mobile robot. The robot guides the user through the process, allowing doctors to remotely monitor and evaluate the results, thus enabling a convenient and localized health assessment process. This mobile robot is highly portable, allowing for short-term deployment in hospital settings as well as long-term use in home environments, adapting to diverse usage needs.

[0126] This system allows doctors to initiate functional testing tasks to service recipients via a remote control platform. Service recipients operate a mobile robot according to the task requirements, and the robot displays relevant information or guides them through self-checks based on the task type. Doctors provide real-time guidance and conduct consultations via remote video and audio interfaces, and the system feeds back the results to the doctor for analysis. This function can be widely applied in telemedicine scenarios such as chronic disease management, postoperative follow-up, and rehabilitation assessment, improving service coverage and resource allocation efficiency.

[0127] The specific steps are as follows:

[0128] Step S101: Doctors select the task type (such as cognitive assessment or limb training) through the remote control platform, and can remotely control the movement and interactive interface of the mobile robot in real time, and issue task instructions.

[0129] Step S102: Enter local mode, the task instruction is sent to the mobile robot, the mobile robot autonomously guides the user to execute, and the mobile robot autonomously follows the service object to move.

[0130] Figure 6 The flowchart of the mobile robot's autonomous following process is shown, which can be divided into three steps: target human body tracking, human body position estimation, and robot following control.

[0131] Step S1021: In the target human body tracking stage, the service object is tracked using the RGB image captured by the camera to obtain the human body anchor frame and skeletal joints.

[0132] Specifically: Based on the RGB images captured by the RGB camera, the existing target detection model is used to perform target detection and keypoint recognition, and the anchor boxes representing the location of the service object and their corresponding human skeletal keypoints in the RGB image are obtained, thereby tracking the target ID.

[0133] Step S1022: In the human body position estimation stage, the coordinates of the service object in space are estimated based on the skeletal joints and the depth image (i.e., the depth image).

[0134] Specifically:

[0135] (1) First, calculate the geometric center of the effective joints of the human skeleton.

[0136] Human skeletal joints In this context, N represents the number of joints. Let be the coordinates of the i-th joint; unidentified joints are called invalid joints, and their coordinate values ​​are all 0. The identified joints are considered valid joints.

[0137] Extract the set of valid joints from all skeletal joints. And calculate the pixel coordinates of the geometric center of the effective joint. The pixel coordinates of the target service object in the pixel coordinate system are represented by the pixel coordinates of the geometric center. ; .

[0138] (2) Calculate the position coordinates of the target service object in the camera coordinate system based on the depth image captured by the RGB camera and the pixel coordinates of the geometric center;

[0139] Specifically, depth information corresponding to the pixel coordinates of the geometric center is obtained from the depth image. This represents the distance from the target service object to the camera along the Z-axis; then the geometric center pixel coordinates are transferred from the pixel coordinate system. Mapped to camera coordinate system .

[0140] (3) Finally, the coordinate estimates of the target service object are obtained by using a Kalman filter through two steps: prediction and correction.

[0141] Step S1023: In the robot following control stage, a PID (Proportion Integration Differentiation) controller is used to calculate the linear velocity and angular velocity required for the robot to move during following control.

[0142] Specifically, based on the target object's position coordinates in the camera coordinate system, the mobile robot's target following position is calculated, thereby obtaining the mobile robot's linear and angular velocities, and controlling the mobile robot's movement. During the following process, the mobile robot can continuously track the target object and maintain follow-up without losing it even when there are multiple people in the scene.

[0143] like Figure 7 As shown, specifically: the current moment is calculated based on the pose changes of the target service object and the pose changes of the mobile robot in the previous frame. Calculate the target pose of the mobile robot and its linear velocity. With angular velocity This enables the mobile robot to move in tandem with the target service object.

[0144] (1) For calculation Time linear velocity , needs to be calculated target location at any time Current location of the mobile robot Distance between This is used as the distance error for controlling the linear velocity. Then, in the camera coordinate system of the mobile robot, the current position of the mobile robot is... Set it to 0.

[0145] During the following process, the mobile robot must maintain a set distance from the target service object. Considering that the mobile robot needs to face the target service object, a distance from the target service object is selected on the line segment between the coordinates of the mobile robot and the target service object. If the point is taken as the coordinate point of the target position of the mobile robot, then:

[0146] ;

[0147] in, The X-axis and Y-axis coordinates of the target service object in the camera coordinate system.

[0148] (2) For calculation angular velocity at time It requires calculation target angle at any time Current angle of the mobile robot angular error To control the angular velocity; then, in the camera coordinate system of the mobile robot, the current angle... Since the value is always 0, the angle error is the target angle, i.e. .

[0149] The strategy involves a following approach where the mobile robot always faces the target service object; the target angle is the angle difference between the target position and the direction the mobile robot is facing. It can be calculated based on the coordinates of the target service object in the camera coordinate system of the mobile robot. :

[0150] ;

[0151] in, It is the arctangent function. The X-axis and Z-axis coordinates of the target service object in the camera coordinate system.

[0152] (3) Calculated and Then, they can be calculated separately. linear velocity at time t With angular velocity :

[0153] ;

[0154] ;

[0155] in, It is proportional gain. It is integral gain. It is the differential gain; for angular error at any given time; for Distance error at any given time; The angle error at time t-1.

[0156] S103: Based on tasks assigned by doctors, the mobile robot guides users to complete medical interaction tasks through voice and screen prompts. The mobile robot is equipped with a toolset that allows users to perform related interaction tasks, including functional testing, cognitive ability assessment, limb rehabilitation training, basic vital sign monitoring, visual and auditory response assessment, MMSE scale testing, and limb rehabilitation exercises. Doctors can also conduct remote consultations with users to inquire about their medical conditions.

[0157] S104: Special screening for cognitive impairment: The mobile robot initiates a cognitive assessment program (such as memory games and attention tests) and records user interaction behavior; relevant medical data of the service recipients (such as action completion, vital signs, and reaction time) are collected in real time through a set of tools and devices, and a preliminary assessment report is generated in combination with the test results.

[0158] S105: The toolset synchronously transmits data to a remote control platform for doctors to monitor. Doctors and service recipients can interact via voice (service recipients can collaborate locally and remotely). Reports are automatically pushed to the doctor's end, and the doctor can remotely revise the assessment conclusions or trigger secondary tests.

[0159] S106: Remote Mode Intervention and Adjustment: Doctors can view the data sent to the server by the mobile robot in real time through the platform. If the user's operation deviates from expectations (such as incorrect rehabilitation movements), the doctor switches to remote control to directly adjust the mobile robot's movements or task flow.

[0160] S107: Records all intervention data for subsequent rehabilitation program optimization. All data is encrypted and stored in the cloud, supporting cross-device access and long-term health trend analysis.

[0161] It should be noted that all data acquisition is conducted in accordance with laws and regulations and with user consent, and the data is used legally.

[0162] Example 2

[0163] This embodiment provides a dual-mode operation control system for a mobile robot, including:

[0164] The redirection module is configured to acquire the user's spatial pose data sent by the remote end in remote mode, determine whether it exceeds the boundary based on the distance between the user and the set walkable area boundary, and if it exceeds the boundary, trigger redirection, calculate the user's rotation direction and distance, until the user faces the center of the walkable area.

[0165] The remote control module is configured to obtain the rotation error based on the rotation increment of the mobile robot and the user at the current moment, obtain the position error based on the movement distance increment, and obtain the angular velocity and linear velocity of the mobile robot based on the rotation error and the position error, so as to control the movement of the mobile robot according to the user's movement.

[0166] The local control module is configured to, in local mode, identify skeletal joints in the acquired image of the service object based on the task received from the remote end, thereby calculating the position of the service object in the camera coordinate system, and calculating the linear and angular velocities of the mobile robot based on the position changes of the service object, so as to control the mobile robot to follow the service object.

[0167] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0168] In further embodiments, the following is also provided:

[0169] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0170] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0171] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0172] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0173] The method in Example 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0174] A computer program product includes a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0175] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0176] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0177] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0178] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0179] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A mobile robot dual-mode operation control method, characterized by, Interactions between remote endpoints, mobile robot endpoints, and service recipient endpoints include: In remote mode, the spatial pose data of the user sent by the remote end is obtained. Based on the distance between the user and the set boundary of the walkable area, it is determined whether the user has exceeded the boundary. If the user has exceeded the boundary, redirection is triggered, and the user's rotation direction and distance are calculated until the user is facing the center of the walkable area. After triggering the redirection, the movement and view of the mobile robot are frozen. Based on the user's current orientation, the user's current position coordinates, and the center coordinates of the walkable area, the angle between the direction vector of the user's current orientation and the direction vector of the user's orientation toward the center point of the walkable area is obtained and displayed in text form on the remote end to guide the user to rotate and walk in the center of the walkable area. At the same time, the angle is compared with a preset threshold. When the angle is less than the preset threshold, it is determined that the user is heading toward the center point of the walkable area. The rotation error is obtained based on the rotation increment of the mobile robot and the user at the current moment, the position error is obtained based on the movement distance increment, and the angular velocity and linear velocity of the mobile robot are obtained based on the rotation error and the position error, so as to control the movement of the mobile robot according to the user's movement. In local mode, based on the task received from the remote end, the robot identifies the skeletal joints of the acquired service object image to calculate the position of the service object in the camera coordinate system. Based on the position change of the service object, the robot calculates the linear velocity and angular velocity of the mobile robot to control the mobile robot to follow the service object.

2. The dual-mode operation control method for a mobile robot as described in claim 1, characterized in that, According to the difference between the rotation angle of the mobile robot end at the current time and at the previous time, a rotation increment is obtained , According to the difference between the rotation angle of the user at the current time and at the previous time, a rotation increment is obtained , The rotation error at the current time is: ; The rotation error at t-1 time The incremental distance is calculated based on the difference between the distance traveled by the mobile robot at the current moment and the distance traveled at the previous moment. The increment of the travel distance is obtained based on the difference between the user's travel distance at the current time and the previous time. Position error at the current moment for: ; The position error at time t-1.

3. The dual-mode operation control method for a mobile robot as described in claim 2, characterized in that, angular velocity of the moving robot at the current moment and linear velocity They are respectively: ； ； in, It is proportional gain. It is integral gain. It is the differential gain; for Rotation error at any given time; for Positional error at any given time.

4. The dual-mode operation control method for a mobile robot as described in claim 1, characterized in that, After identifying skeletal joints, a set of valid joints is extracted, and the pixel coordinates of the geometric center of the valid joints are used as the coordinates of the service object in the pixel coordinate system. Based on the depth information corresponding to the pixel coordinates in the acquired depth image, the coordinates of the service object in the pixel coordinate system are mapped to the coordinates in the camera coordinate system of the mobile robot.

5. The dual-mode operation control method for a mobile robot as described in claim 1, characterized in that, The process of calculating the linear velocity and angular velocity of a mobile robot based on the positional changes of the service object includes: Depend on target location at any time Current location of the mobile robot Distance between As control Time linear velocity Distance error Then, in the camera coordinate system of the mobile robot, the current position of the mobile robot is... Setting it to 0 means that the mobile robot maintains a set distance from the service recipient during the following process. Select a distance between the mobile robot and the service recipient. The point is taken as the target position of the mobile robot, and thus ;in, The X-axis and Y-axis coordinates of the service object in the camera coordinate system; Depend on target angle at any time Current angle of the mobile robot angular error right angular velocity at time To perform control, the current angle in the camera coordinate system of the mobile robot is... It is always 0, so Furthermore, the target angle is the angle difference between the target position and the orientation of the mobile robot. Therefore, based on the position coordinates of the service object in the camera coordinate system, we can obtain... : ;in, It is the arctangent function. The X-axis and Z-axis coordinates of the target service object in the camera coordinate system; linear velocity at time t With angular velocity They are respectively: ； ； in, It is proportional gain. It is integral gain. It is the differential gain; for angular error at any given time; for Distance error at any given time; The angle error at time t-1.

6. A dual-mode operation control system for a mobile robot employing the dual-mode operation control method for a mobile robot as described in any one of claims 1-5, characterized in that, include: The redirection module is configured to acquire the user's spatial pose data sent by the remote end in remote mode, determine whether it exceeds the boundary based on the distance between the user and the set walkable area boundary, and if it exceeds the boundary, trigger redirection, calculate the user's rotation direction and distance, until the user faces the center of the walkable area. The remote control module is configured to obtain the rotation error based on the rotation increment of the mobile robot and the user at the current moment, obtain the position error based on the movement distance increment, and obtain the angular velocity and linear velocity of the mobile robot based on the rotation error and the position error, so as to control the movement of the mobile robot according to the user's movement. The local control module is configured to, in local mode, identify skeletal joints in the acquired image of the service object based on the task received from the remote end, thereby calculating the position of the service object in the camera coordinate system, and calculating the linear and angular velocities of the mobile robot based on the position changes of the service object, so as to control the mobile robot to follow the service object.

7. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-5.

9. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-5.

Images