Hybrid control method and device for mobile operation robot, robot and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to the medical field and the financial science and technology field, and discloses a mobile operation robot hybrid control method and device, a robot and a medium, and the method comprises the steps: collecting a multi-view RGB image and the body state data of the robot; performing convolution feature extraction on the RGB image of each view angle to obtain a feature map of each view angle, performing feature fusion on the cross-view-angle feature map to generate global scene representation, and generating a target pose of the end effector in a world coordinate system; calculating the expected speed of the end effector in the task space; the base speed and the mechanical arm joint speed are solved in real time, and a target speed instruction is obtained; and the target speed instruction is sent to a bottom layer actuator of the robot so as to drive the robot to complete a motion task, and the state is fed back to the sensing and control module in real time. The multi-view perception information is effectively integrated, and the decision stability is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and can be applied to the medical and financial technology fields. In particular, it relates to a hybrid control method, device, robot, and medium for a mobile operating robot. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, the autonomous operation capability of mobile robots in unstructured environments has become an important research direction in industrial, service, and specialized fields. However, existing technologies still suffer from insufficient perception robustness, control jitter, and unstable decision-making when facing dynamic and complex scenarios, limiting their widespread application in fields with higher reliability requirements.

[0003] In the fintech sector, automated services and security patrols place stringent demands on robots' perception and decision-making capabilities. Bank service robots need to identify customers, deliver documents, and assist in operations within complex lobby environments, while avoiding collisions with people and facilities. However, existing systems often rely on single-view cameras and predetermined path planning, making them prone to losing targets in crowded areas or with visual obstructions, leading to service interruptions. In the medical assistance and rehabilitation nursing fields, the application scenarios for mobile operating robots place extremely high demands on system stability and security. Ward service robots need to perform tasks such as delivering medication, delivering instruments, and assisting patients in repositioning. Their operating environment contains numerous dynamic obstacles, such as medical staff, beds, and equipment, and there are strict limitations on the smoothness and safety of movement. Existing robot systems often separate navigation and operation in their design, resulting in slow adjustments and inconsistent trajectories when approaching beds or operating tables, and potentially causing medical risks due to control jitter.

[0004] Currently, the industry is attempting to improve perception and decision-making capabilities by introducing deep learning models. For example, target detection networks are used to track people in financial security, or reinforcement learning is used for path planning in medical robots. However, these methods often experience sudden decision changes or discontinuous control in real-world scenarios due to environmental interference and data distribution shifts. Furthermore, the lack of systematic modeling of physical constraints leads to execution-level jitter and compromised security. In particular, existing technologies fail to effectively integrate multi-view perception information to enhance scene understanding and do not fully consider system constraints in control strategies, resulting in unreliable and unsmooth robot operation in complex environments. Summary of the Invention

[0005] The purpose of this application is to propose a hybrid control method, device, robot, and medium for mobile operating robots, which has the advantages of effectively integrating multi-view perception information, improving decision-making stability, ensuring control smoothness, and reducing jitter.

[0006] To address the aforementioned technical problems, embodiments of this application provide a hybrid control method for a mobile operating robot, comprising: Multi-view RGB images are simultaneously acquired by multiple cameras deployed on the robot, and the robot's body state data is also acquired simultaneously. Convolutional feature extraction is performed on the RGB images from each viewpoint to obtain feature maps for each viewpoint, and feature fusion is performed on the feature maps across viewpoints to generate a global scene representation. The global scene representation and the robot's current state are encoded into a conditional vector and input into a pre-trained diffusion policy network to generate the target pose of the end effector in the world coordinate system. The expected velocity of the end effector in the task space is calculated based on the error between the current end effector pose and the target pose. Based on the desired speed, the current state of the robot, and system constraints, the base speed and the joint speed of the robotic arm are solved in real time to obtain the target speed command; The target speed command is sent to the robot's underlying actuator to drive the robot to complete the motion task and to provide real-time feedback on the status to the perception and control module.

[0007] To address the aforementioned technical problems, embodiments of this application provide a hybrid control device for a mobile operating robot, comprising: The data acquisition module is used to simultaneously acquire RGB images from multiple perspectives using multiple cameras deployed on the robot, and to simultaneously acquire the robot's body state data. The feature extraction module is used to perform convolutional feature extraction on the RGB images from each viewpoint to obtain feature maps for each viewpoint, and to perform feature fusion on the feature maps across viewpoints to generate a global scene representation. The target pose generation module is used to encode the global scene representation and the robot's current state into a conditional vector, and input it into a pre-trained diffusion policy network to generate the target pose of the end effector in the world coordinate system. The expected velocity calculation module is used to calculate the expected velocity of the end effector in the task space based on the error between the current end effector pose and the target pose. The speed command generation module is used to solve the base speed and the robot joint speed in real time based on the desired speed, the current state of the robot and system constraints, so as to obtain the target speed command. The motion task driving module is used to send the target speed command to the robot's underlying actuator to drive the robot to complete the motion task and to provide real-time feedback on the status to the perception and control module.

[0008] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: a mobile operation robot, comprising: Multiple cameras are used to simultaneously capture RGB images of the environment from different perspectives; The robot body includes a movable chassis, a robotic arm, and an end effector; A status sensor is used to acquire the position and orientation of the chassis, the joint state of the robotic arm, and the pose of the end effector. The controller includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the hybrid control method for the mobile operating robot described above.

[0009] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the hybrid control method for mobile operation robots described in any one of the above-mentioned methods.

[0010] This invention provides a hybrid control method, device, robot, and medium for a mobile robot. The method includes: simultaneously acquiring multi-view RGB images from multiple cameras deployed on the robot, and simultaneously acquiring the robot's body state data; performing convolutional feature extraction on the RGB images from each viewpoint to obtain feature maps for each viewpoint, and fusing the feature maps across viewpoints to generate a global scene representation; encoding the global scene representation and the robot's current state into a conditional vector, and inputting it into a pre-trained diffusion policy network to generate a target pose of the end effector in the world coordinate system; calculating the expected velocity of the end effector in the task space based on the error between the current end effector pose and the target pose; solving the base velocity and the robotic arm joint velocity in real time based on the expected velocity, the robot's current state, and system constraints to obtain a target velocity command; sending the target velocity command to the robot's underlying actuator to drive the robot to complete the motion task, and providing real-time status feedback to the perception and control module. This invention addresses the problems of insufficient perception robustness, unstable decision-making, and control jitter in existing technologies by simultaneously acquiring multi-view images, fusing features to generate a global scene representation, and using a diffusion strategy network to generate target poses. It also considers system constraints by combining real-time speed solving. This has the advantages of effectively integrating multi-view perception information, improving decision-making stability, ensuring control smoothness, and reducing jitter. Attached Figure Description

[0011] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of an application environment for a hybrid control method for a mobile operating robot according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the implementation of the hybrid control method for mobile operating robots provided in this application embodiment; Figure 3 yes Figure 2 A flowchart illustrating a specific implementation method of step S2; Figure 4 yes Figure 2 A flowchart illustrating a specific implementation method of step S3; Figure 5 yes Figure 3 A flowchart illustrating a specific implementation of step S31; Figure 6 yes Figure 3 A flowchart illustrating a specific implementation of step S32; Figure 7 yes Figure 2 A flowchart illustrating a specific implementation of step S4; Figure 8 This is a schematic diagram of the hybrid control device for a mobile operating robot provided in an embodiment of this application; Figure 9 This is a schematic diagram of the mobile operation robot provided in the embodiments of this application. Detailed Implementation

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0014] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0016] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0017] It should be noted that the mobile operation robot hybrid control method provided in this application embodiment is generally executed by the robot, and correspondingly, the mobile operation robot hybrid control device is generally configured in the robot.

[0018] The hybrid control method for mobile operating robots provided in this invention can be applied to, for example... Figure 1 In this application environment, the user communicates with the robot via a network. The robot can receive instructions from the user and execute movement tasks accordingly. The invention will now be described in detail through specific embodiments.

[0019] The hybrid control method for mobile operating robots provided in this application can be applied to the intelligent management and delivery of sterile instruments in operating rooms, or to intelligent guidance and service robots in bank branches.

[0020] Traditional mobile robots often face challenges such as insufficient perception robustness, control jitter, and unstable decision-making when operating autonomously in unstructured environments. For example, in automated services and security patrols in the fintech sector, existing systems rely heavily on single-view cameras and predetermined paths, making them prone to losing targets in crowded areas or when visual obstruction occurs, leading to service interruptions. In the fields of medical assistance and rehabilitation care, existing robot systems often separate navigation and operation, resulting in slow adjustments and discontinuous trajectories when approaching the control panel, and potentially causing medical risks due to control jitter. While the industry has attempted to introduce deep learning models, issues such as environmental interference and data distribution shifts frequently lead to abrupt decision changes or discontinuous control, and the lack of systematic modeling of physical constraints results in execution-level jitter, making it difficult to guarantee safety.

[0021] To address this, this application proposes a hybrid control method for mobile robots, comprising: synchronously acquiring RGB images from multiple perspectives using multiple cameras deployed on the robot, and simultaneously acquiring the robot's body state data; performing convolutional feature extraction on the RGB images from each perspective to obtain feature maps for each perspective, and fusing features from the cross-perspective feature maps to generate a global scene representation; encoding the global scene representation and the robot's current state into a conditional vector, and inputting it into a pre-trained diffusion policy network to generate the target pose of the end effector in the world coordinate system; calculating the expected velocity of the end effector in the task space based on the error between the current end effector pose and the target pose; solving the base velocity and the manipulator joint velocity in real time based on the expected velocity, the robot's current state, and system constraints to obtain the target velocity command; sending the velocity command to the robot's underlying actuator to drive the robot to complete the motion task, and providing real-time feedback of the state to the perception and control module.

[0022] For ease of understanding, the following explains some key terms in this embodiment: Mobile manipulation robot: A robotic system that integrates a mobile chassis and a robotic arm, capable of moving in the environment and performing manipulation tasks.

[0023] Multi-camera: refers to multiple image acquisition devices deployed on the robot body that can simultaneously acquire environmental visual information from different spatial positions or angles.

[0024] RGB image: Color image data containing the three primary color channels of red, green and blue, used to represent visual information about the environment.

[0025] Body state data: Data describing the robot's own kinematic and dynamic state, such as the position and orientation of the base, and the angles and angular velocities of the joints of the robotic arm.

[0026] Convolutional feature extraction: The process of using convolutional neural networks to process image data and extract semantic and geometric features.

[0027] Feature map: A two-dimensional or multi-dimensional data structure containing local or global feature information of an image, obtained after convolutional feature extraction.

[0028] Feature fusion: The process of integrating feature maps from different perspectives or levels to generate a more comprehensive and robust scene representation.

[0029] Global scene representation: A holistic understanding and abstract description of the robot's environment obtained after feature fusion.

[0030] Conditional vectors encode various input information (such as scene representation and robot state) into a unified numerical vector form, which serves as the input conditions for deep learning models.

[0031] Pre-trained diffusion policy network: A deep learning network built on the principle of diffusion model, which is pre-trained on a large amount of data and can generate outputs that conform to a specific distribution (such as robot end pose) according to given conditions.

[0032] End effector: The foremost part of a robotic arm used to interact with the environment, such as grippers, suction cups, or tools.

[0033] World coordinate system: A fixed, global reference coordinate system used to describe the position and orientation of the robot and objects in its environment.

[0034] Target pose: The position and orientation that the end effector is expected to achieve in the world coordinate system.

[0035] Current end effector pose: The current actual position and orientation of the end effector in the world coordinate system.

[0036] Task space: A Cartesian space that describes the motion of the end effector, typically including position and orientation.

[0037] Expected velocity: The linear and angular velocities that the end effector is expected to achieve in the task space.

[0038] System constraints: The physical and operational limitations that a robot system must adhere to during its movement, such as the range of motion of joints, speed limits, base movement speed limits, and requirements to avoid collisions with the environment.

[0039] Base speed: The linear and angular speeds of the moving robot chassis.

[0040] Robotic arm joint velocity: the angular velocity of each joint of the robotic arm.

[0041] Target speed command: The final speed command, calculated by the control algorithm, is sent to the robot's underlying actuators to drive their movement.

[0042] Low-level actuator: The hardware unit inside the robot that directly drives the motors, joints and other components to perform motion.

[0043] Perception and Control Module: The core software or hardware unit in a robot system responsible for processing sensor data, understanding the environment, making decisions and planning, and controlling motion.

[0044] This embodiment provides a hybrid control method for a mobile operating robot.

[0045] First, multiple cameras deployed on the robot simultaneously acquire multi-view RGB images and obtain the robot's own state data. Specifically, the mobile robot can be equipped with multiple vision sensors, such as cameras, which are fixed at different positions on the robot body to cover a larger spatial range. During the robot's task execution, these vision sensors are configured to capture RGB images of the environment at the same time, ensuring that the acquired image data remains consistent over time. Simultaneously, sensors inside the robot, such as encoders and inertial measurement units, are used to monitor and record the motion information of the robot's base and the current angles of each joint of the robotic arm in real time. This body state data is acquired synchronously with the visual data to provide accurate information about the robot's own posture.

[0046] Furthermore, convolutional feature extraction is performed on the RGB images from each viewpoint to obtain feature maps for each viewpoint. Feature fusion is then performed on the cross-viewpoint feature maps to generate a global scene representation. Specifically, the RGB images from each viewpoint acquired from multiple cameras can be processed separately using independent image processing units. For example, each viewpoint image is input into a network composed of multiple convolutional and pooling layers to extract different levels of visual information from the image, thereby generating corresponding feature maps. Subsequently, to integrate these local features from different viewpoints, these feature maps can be simply stitched together, or combined using element-wise summation, averaging, or other methods. In this way, the feature information from each viewpoint is aggregated to form a global scene representation that reflects the overall situation of the robot's surrounding environment.

[0047] Building upon this, the global scene representation and the robot's current state are encoded into a conditional vector, which is then input into a pre-trained diffusion policy network to generate the target pose of the end effector in the world coordinate system. Specifically, the generated global scene representation is integrated with the robot's current body state data (e.g., base position, robotic arm joint angles, etc.). This information can be concatenated and dimensionality-reduced and encoded using an auxiliary neural network to form an integrated conditional vector. This conditional vector is then input into a pre-trained diffusion policy network. This diffusion policy network is designed to gradually recover the desired target pose of the end effector in the world coordinate system from initial noise through an iterative denoising process, given the environment and robot state.

[0048] Subsequently, the desired velocity of the end effector in the task space is calculated based on the error between the current end effector pose and the target pose. Specifically, after obtaining the target pose of the end effector, it needs to be compared with the robot's current actual end effector pose. The current end effector pose can be calculated through robot forward kinematics or directly obtained from the end effector sensors. By calculating the position and orientation differences between the target pose and the current pose, a pose error can be obtained. This pose error is then directly converted into the desired linear and angular velocities of the end effector in the task space. For example, the components of the pose error can be multiplied by a set scaling factor to determine the velocity at which the end effector should move towards the target pose.

[0049] Furthermore, based on the desired velocity, the robot's current state, and system constraints, the base velocity and the robot arm joint velocities are calculated in real time to obtain the target velocity command. Specifically, the calculated desired velocity of the end effector, the robot's current body state (including base position, posture, robot arm joint angles, etc.), and various system constraints that the robot must adhere to during operation (e.g., maximum rotational speed of each joint, maximum base movement speed, and general requirements to avoid collisions with the environment) are used as inputs. An inverse kinematics or optimization controller is used to process these inputs in real time. Through iterative calculations, this controller determines the linear and angular velocities of the robot base and the angular velocities of each joint of the robot arm, while satisfying all system constraints, so that the end effector can move at the desired velocity as much as possible. This generates the target velocity command used to drive the robot's motion.

[0050] Finally, the target speed command is sent to the robot's underlying actuators to drive the robot to complete the motion task and provides real-time feedback to the perception and control module. Specifically, the generated target speed command is transmitted to the motor drivers of the robot chassis and the servo controllers of each joint of the robotic arm via the robot's internal communication bus or network interface. Upon receiving the command, these underlying actuators immediately drive the corresponding motors and joints to move at the specified speed, thereby enabling the robot to complete the set operation task. During the robot's movement, its internal sensors continuously monitor the actual motion status of the base and robotic arm and feed this real-time status data back to the perception and control module. This closed-loop feedback process ensures that the robot can adjust its movement according to the actual situation to cope with environmental changes or execution errors.

[0051] This application addresses the issue of insufficient perception robustness of mobile robots in dynamic and complex environments by integrating multi-camera synchronous perception, cross-view feature fusion, and decision generation based on diffusion strategy networks. By solving the coordinated speed of the base and robotic arm in real time and optimizing it under system constraints, control jitter is reduced, and the smoothness and safety of motion are improved. Therefore, this method enables robots to achieve stable, continuous, and safe autonomous operation in scenarios with high reliability requirements, such as financial security and medical assistance, avoiding problems such as target loss, discontinuous trajectories, or abrupt decision changes.

[0052] Please see Figure 2 , Figure 2 A specific implementation of a hybrid control method for a mobile operating robot is shown.

[0053] It should be noted that if substantially the same result is obtained, the method of this invention is not based on... Figure 2 Limited to the order of the processes shown, this method includes the following steps: S1: Simultaneously acquire RGB images from multiple perspectives using multiple cameras deployed on the robot, and simultaneously obtain the robot's body state data.

[0054] Please see Figure 3 , Figure 3 A specific implementation of step S1 is shown below: S11: Simultaneously acquire RGB images from multiple perspectives at the same time using multiple cameras deployed on the robot. S12: Simultaneously acquire the robot's body state data using state sensors, wherein the body state data includes the base position and orientation, the robotic arm joint state, and the current pose of the end effector.

[0055] In some of the solutions mentioned above in this application, it is proposed to synchronously acquire RGB images from multiple perspectives using multiple cameras deployed on the robot and synchronously obtain body state data to provide environmental perception and robot state input. However, in its implementation, there is a lack of a specific mechanism for strictly synchronizing data acquisition, which may lead to time inconsistency between multi-view images and state data, resulting in misalignment of perception information, affecting the accuracy of subsequent feature fusion and control decisions, and thus causing trajectory jitter or discontinuous operation.

[0056] In response, this application further proposes to simultaneously acquire RGB images from multiple perspectives using multiple cameras deployed on the robot, and simultaneously obtain the robot's body state data, including: simultaneously acquiring RGB images from multiple perspectives at the same time using multiple cameras deployed on the robot; and simultaneously obtaining the robot's body state data using state sensors, wherein the body state data includes the base position and attitude, the robotic arm joint state, and the current pose of the end effector.

[0057] Specifically, the multi-camera refers to multiple independent image acquisition devices deployed on the robot body, such as multiple RGB cameras, stereo cameras, or depth cameras. They are installed in a specific spatial layout (such as a surround, front-mounted, or side-mounted combination) to capture environmental information around the robot from different perspectives, providing more comprehensive visual perception input. Synchronous acquisition at the same moment means ensuring that all multi-cameras simultaneously expose images and read data within an extremely short time window (e.g., microseconds), thus guaranteeing that the acquired multi-view RGB images reflect the state of the environment at the same instant. This can be achieved through hardware-triggered synchronization mechanisms, such as all cameras sharing an external synchronization signal source, or through high-precision timestamp alignment and software interpolation.

[0058] The state sensor refers to a device used to monitor and acquire the robot's own kinematic and dynamic state in real time, such as an inertial measurement unit (IMU), joint encoder, lidar, or visual odometry. Synchronous acquisition means that the data acquisition from the state sensor and the image acquisition from multiple cameras are strictly consistent in time, ensuring that the body state data and the visual perception data have the same time reference. This can be achieved through a unified sensor data bus, high-precision timestamp fusion, or hardware interrupt triggering.

[0059] The body state data includes the base position and attitude, the robot arm joint states, and the current pose of the end effector. The base position and attitude refer to the three-dimensional position and attitude information of the mobile robot chassis in the world coordinate system, typically obtained by fusing odometry, IMU, and / or SLAM (Simultaneous Localization and Mapping) algorithms, and are the foundation for the robot's global navigation and localization. The robot arm joint states refer to the real-time angle or position information of each joint of the robot arm, usually measured directly by the encoder inside the robot arm, and are key inputs for robot arm motion control and inverse kinematics calculations. The current pose of the end effector refers to the three-dimensional position and attitude of the robot arm's end tool (such as a gripper, suction cup, etc.) in the world coordinate system. This is usually obtained through forward kinematics calculations of the robot arm, combined with the base position and attitude information, and is the direct control target for performing specific operational tasks.

[0060] This application embodiment synchronously acquires multi-view RGB images from multiple cameras deployed on the robot at the same time, and simultaneously obtains the robot's body state data, ensuring strict temporal consistency between visual perception data and robot body state data. This effectively solves the problem of perception information misalignment caused by inconsistent data timing, avoiding errors introduced by time deviations in subsequent feature fusion and control decisions, thereby significantly improving the accuracy and consistency of perception input. Simultaneously, the body state data clearly includes the base position and posture, the robotic arm joint state, and the current pose of the end effector, ensuring that the state information of all key moving parts of the robot is comprehensively and in real-time acquired. This provides reliable and accurate basic data for subsequent global scene representation, target pose generation, and hybrid control of the base and robotic arm, thereby ensuring the stability, accuracy, and safety of the mobile robot when performing motion tasks, effectively avoiding problems such as trajectory jitter and discontinuous operation.

[0061] S2: Perform convolutional feature extraction on the RGB images from each viewpoint to obtain feature maps for each viewpoint, and perform feature fusion on the feature maps across viewpoints to generate a global scene representation.

[0062] Please see Figure 4 , Figure 4 A specific implementation of step S2 is shown below: S21: Extract high-dimensional semantic and geometric features of the RGB images from each viewpoint using a pre-trained convolutional neural network to obtain feature maps for each viewpoint. S22: Flatten and stitch the feature maps from each viewpoint to generate stitched features. S23: Calculate the correlation weights between the stitched features from each viewpoint using a multi-head self-attention mechanism, and fuse the stitched features based on these correlation weights to obtain weighted fused features. S24: Aggregate the weighted fused features to generate the global scene representation.

[0063] In some of the embodiments described above in this application, feature fusion is proposed to generate a global scene representation to provide an environment representation. However, in its implementation, the multi-view feature fusion method may be inconclusive or not robust, which may lead to insufficient integration of feature information in dynamic and complex scenes or when visual occlusion occurs, resulting in perceptual bias and thus affecting the stability and accuracy of subsequent control decisions.

[0064] To address this, this application further proposes to extract convolutional features from RGB images at each viewpoint to obtain feature maps for each viewpoint, and to fuse the feature maps across viewpoints to generate a global scene representation. Specifically, this includes: extracting high-dimensional semantic and geometric features from the RGB images at each viewpoint using a pre-trained convolutional neural network to obtain feature maps for each viewpoint; flattening and stitching the feature maps at each viewpoint to generate stitched features; calculating the correlation weights between the stitched features at each viewpoint using a multi-head self-attention mechanism, and fusing the stitched features based on the correlation weights to obtain weighted fused features; and aggregating the weighted fused features to generate the global scene representation.

[0065] Specifically, high-dimensional semantic and geometric features of the RGB images from each viewpoint are extracted using a pre-trained convolutional neural network (CNN), resulting in feature maps for each viewpoint. The pre-trained CNN is a deep learning model trained on large image datasets, learning general visual features and possessing powerful feature extraction and generalization capabilities. Its function is to capture rich visual information from the original RGB images, including abstract semantic information such as the category and attributes of objects in the image, as well as spatial geometric information such as the shape, position, and depth of objects. In one implementation, classic pre-trained CNN architectures such as ResNet, VGG, or EfficientNet can be used as feature extractors. In another implementation, Transformer-based visual models such as Vision Transformer (ViT) or Swin Transformer can be used. These models excel in handling global dependencies in images and can also achieve powerful feature extraction capabilities through large-scale pre-training.

[0066] Subsequently, the feature maps from each viewpoint are flattened and stitched together to generate stitched features. This step converts the two-dimensional or three-dimensional feature maps extracted from different viewpoints into one-dimensional vectors and connects them in a specific order to form a unified, longer feature vector, which can then be processed by the subsequent attention mechanism. In one implementation, the spatial dimensions (e.g., height and width) of each feature map can be flattened into a one-dimensional vector, and then the flattened feature vectors from all viewpoints can be stitched together along the channel or feature dimension. In another implementation, each feature map can first undergo global average pooling or max pooling to reduce its dimensionality to a fixed-length vector, and then these vectors can be stitched together.

[0067] Based on this, a multi-head self-attention mechanism is used to calculate the correlation weights between the concatenated features from different perspectives. The concatenated features are then fused based on these correlation weights to obtain weighted fused features. The multi-head self-attention mechanism is a core component of the Transformer model. It allows the model to simultaneously focus on information at different positions in the sequence when processing sequential data, assigning different "attention" to each position. By executing multiple self-attention calculations in parallel, each "head" learns different attention patterns, thereby capturing richer dependencies. Its role is to dynamically learn and assign importance weights between features from different perspectives, thus highlighting key information, suppressing redundant or noisy information, and quantifying the degree of correlation between features from different perspectives during fusion, providing a basis for subsequent weighted fusion. One implementation uses a multi-head self-attention module from the standard Transformer encoder layer. The query, key, and value are all derived from the concatenated features. The relevance weights are obtained by calculating the dot product of the query and key, scaling it, and then normalizing it using a softmax function. The concatenated features from each perspective are then multiplied by their corresponding relevance weights, and all weighted features are summed. Another implementation uses a variant of the cross-attention mechanism. Features from one perspective are used as the query, while features from other perspectives are used as keys and values, enabling more refined cross-perspective information interaction. A gating mechanism is employed, for example, a gating network dynamically adjusts the contribution ratios of features from different perspectives based on their relevance weights before fusion.

[0068] Finally, the weighted fused features are aggregated to generate the global scene representation. This aggregation step further processes the weighted fused features, typically by reducing their dimensionality or transforming them into a compact, fixed-length vector as the final global scene representation. This global scene representation is a compact, high-dimensional vector that encodes overall information about the robot's environment, including objects, layout, and operable areas, serving as the basis for the robot's decision-making and planning. In one implementation, the weighted fused features can be processed using one or more fully connected layers (MLPs) to map them to the desired dimension, thus obtaining the global scene representation. In another implementation, the [CLS] token mechanism from the Transformer encoder can be used. A learnable [CLS] token is added when concatenating features; after passing through a multi-head self-attention layer, the output of this [CLS] token serves as the aggregated global scene representation.

[0069] This application addresses the robustness issue in multi-view feature fusion by specifying detailed feature fusion steps, thereby improving perception accuracy and decision stability in complex environments. First, a pre-trained convolutional neural network extracts high-dimensional semantic and geometric features from the RGB images of each viewpoint. This step leverages the generalization ability of the pre-trained model to efficiently capture semantic and spatial information in the image, ensuring the quality of the feature base and avoiding inconsistencies in feature extraction due to viewpoint differences. Second, the feature maps from each viewpoint are flattened and stitched together. This operation integrates scattered multi-view information into a unified format, facilitating subsequent processing, solving the problem of information isolation between viewpoints, and providing structured input for fusion. Next, a multi-head self-attention mechanism is used to calculate the correlation weights between the stitched features from each viewpoint, and the stitched features are fused based on these correlation weights. The multi-head self-attention mechanism adaptively learns the dynamic correlations between features, assigning weights to different viewpoints to highlight important information and suppress noise, effectively handling occlusion or dynamically changing scenes and ensuring the robustness of the fusion process. Finally, the weighted fused features are aggregated to generate a compact global scene representation, providing comprehensive environmental information and supporting accurate input for subsequent control modules. This refined feature fusion mechanism enables robots to understand complex scenes more accurately. Even when some perspectives are obstructed or the environment changes dynamically, it can compensate with effective information from other perspectives, thus providing a more stable and reliable perception basis for subsequent target pose generation and motion control. This significantly improves the operational capabilities and decision-making stability of mobile robots in unstructured environments.

[0070] S3: Encode the global scene representation and the robot's current state into a conditional vector, and input it into a pre-trained diffusion policy network to generate the target pose of the end effector in the world coordinate system.

[0071] Please see Figure 5 , Figure 5 A specific implementation method prior to step S3 is shown below in detail: S31: Collect multi-view teleoperation demonstration data, wherein the teleoperation demonstration data includes multi-view RGB image sequences, robot state sequences, and end-effector trajectories. S32: Add noise to the end-effector trajectory to construct noise-denoising sample pairs. S33: Using the images in the multi-view RGB image sequences and the robot states in the robot state sequences as conditions, train the diffusion strategy network based on the noise and denoising samples to learn the denoising process for recovering the end-effector trajectory from the noise, thus obtaining the trained diffusion strategy network.

[0072] In some of the embodiments described above in this application, a pre-trained diffusion policy network is proposed to generate the target pose of the end effector in the world coordinate system. However, in its implementation, without an effective training method, the diffusion policy network may not be able to accurately learn to recover the end effector trajectory from noise, resulting in unstable decision-making and discontinuous control, thereby affecting the smoothness and safety of the robot's motion in dynamic and complex environments.

[0073] To address this, this application further proposes that before encoding the global scene representation and the robot's current state into a conditional vector and inputting it into a pre-trained diffusion policy network to generate the target pose of the end effector in the world coordinate system, the method further includes the following steps: acquiring multi-view teleoperation demonstration data, wherein the teleoperation demonstration data includes multi-view RGB image sequences, robot state sequences, and end effector trajectories; adding noise to the end effector trajectory to construct noise and denoising sample pairs; using the images in the multi-view RGB image sequences and the robot state in the robot state sequences as conditions, training the diffusion policy network based on the noise and denoising samples to learn the denoising process of recovering the end effector trajectory from the noise, thereby obtaining a trained diffusion policy network.

[0074] Specifically, collecting multi-view teleoperation demonstration data refers to the various sensor data and robot motion data recorded when a robot is remotely controlled by a human to complete a specific task. This data is used to provide demonstrations for robot learning tasks. Multi-view RGB image sequences provide visual information about the environment during task execution, capturing scene details from different angles, which helps the robot understand three-dimensional spatial structures and object relationships. Robot state sequences record information such as the robot's body posture, joint angles, and base position, reflecting the robot's own kinematics and dynamics. The end effector trajectory records the continuous pose changes of the end effector in the world coordinate system, which is the critical path information for the robot to complete the task. One implementation method is to have a human operator remotely control the robot to perform tasks using teleoperation devices (such as force feedback master hands, VR / AR devices), while the robot's own perception system (multi-channel cameras, state sensors) synchronously records the above data. Another implementation method is to conduct expert demonstrations in a simulation environment, using a simulator to generate high-fidelity multi-view RGB images, robot state, and end effector trajectory data.

[0075] Adding noise to the terminal trajectory and constructing noise-and-denoised sample pairs aims to simulate potential perceptual errors, execution errors, or external disturbances in a real-world environment, thereby enhancing the robustness of the diffusion policy network to uncertainties. By adding noise, the original "clean" terminal trajectory can be transformed into a "noisy" trajectory, thus constructing sample pairs consisting of the noisy trajectory and the original trajectory. These sample pairs are used to train the diffusion policy network to learn how to recover the original, smooth terminal trajectory from the noise, i.e., to perform denoising. One implementation is to superimpose random noise, such as Gaussian noise or uniformly distributed noise, onto the pose (position and orientation) of the terminal trajectory at each time step, with its variance or amplitude adjustable according to the noise level of the actual application scenario. Another implementation is to introduce time-dependent noise, such as Brownian motion noise, giving the trajectory noise a certain continuity and dynamism, which better reflects disturbances in actual physical processes.

[0076] Using the images in the multi-view RGB image sequence and the robot states in the robot state sequence as conditions, the diffusion strategy network is trained based on the noise and denoised samples to learn the denoising process for recovering the end-effector trajectory from the noise, resulting in a trained diffusion strategy network. This step aims to train the diffusion strategy network to progressively recover the original, noise-free end-effector trajectory from a noisy end-effector trajectory based on the current visual perception (multi-view RGB images) and the robot's own state. During training, the network receives the noisy trajectory along with the images and robot states as conditions, and predicts the denoised trajectory. By iteratively learning the denoising process, the network understands the influence of the environmental context and the robot's own state on end-effector trajectory generation, thereby generating a reasonable target pose that conforms to the current scene and robot state during inference. One implementation is to use a training framework based on the Denoising Diffusion Probability Model (DDPM), where the diffusion strategy network is implemented as a U-Net neural network. Its input includes the noisy trajectory and time step information, and multi-view RGB image features and robot state features are used as conditional inputs through a cross-attention mechanism or feature concatenation. The network's goal is to predict the noise given the noisy trajectory and conditions, thereby achieving denoising. Another approach is to use the training paradigm of Conditional Generative Adversarial Networks (cGANs), which uses a diffusion policy network as the generator and introduces a discriminator to evaluate the authenticity of the generated trajectories and their matching degree with the conditions. This approach improves the quality and condition consistency of the generated trajectories through adversarial training.

[0077] Before encoding the global scene representation and the robot's current state into conditional vectors and inputting them into a pre-trained diffusion strategy network to generate the target pose of the end effector in the world coordinate system, this application collects multi-view teleoperation demonstration data. This yields rich demonstration information including multi-view RGB image sequences, robot state sequences, and end effector trajectories, providing comprehensive contextual data for training the diffusion strategy network. This enables the network to learn complex perception-action mapping relationships in the real world. By adding noise to the end effector trajectory and constructing noise-denoising sample pairs, this application simulates various uncertainties and disturbances that may exist in actual operation, forcing the network to learn to recover accurate end effector trajectories from these uncertainties, significantly enhancing the network's robustness to environmental disturbances and perceptual noise. Furthermore, using images from the multi-view RGB image sequence and the robot state from the robot state sequence as conditions, the diffusion strategy network is trained based on noise and denoising samples. This allows the network to fully utilize real-time environmental visual information and the robot's own state during the denoising process, ensuring that the generated end effector trajectory is not only smooth and meets task requirements but also highly consistent with the current scene and the robot's own state. Ultimately, through this supervised and robust training process, a well-trained diffusion policy network was obtained. This network can generate the target pose of the end effector more accurately and stably from the sensory input and the robot's state, effectively solving the problems of unstable decision-making and discontinuous control. This significantly improves the motion smoothness, safety, and task success rate of the mobile robot in dynamic and complex environments.

[0078] S4: Calculate the expected speed of the end effector in the task space based on the error between the current end effector pose and the target pose.

[0079] Please see Figure 6 , Figure 6 A specific implementation of step S4 is shown below: S41: Calculate the pose error based on the current end effector pose and the target pose. S42: Calculate the desired velocity of the end effector in the task space based on the proportional gain matrix and the pose error.

[0080] In some of the embodiments described above in this application, the desired velocity of the end effector in the task space is calculated based on the error between the current end effector pose and the target pose in order to control the robot's motion. However, in its implementation, due to the lack of a specific calculation mechanism, it may lead to inaccurate error processing and unstable control response, thereby causing robot motion jitter or unstable decision-making.

[0081] In this regard, this application further proposes a step for calculating the expected speed of the end effector in the task space, including: calculating the pose error based on the current end effector pose and the target pose; and calculating the expected speed of the end effector in the task space based on the proportional gain matrix and the pose error.

[0082] Specifically, calculating the pose error based on the current end effector pose and the target pose refers to quantifying the spatial deviation between the actual pose of the robot's end effector and the desired target pose. This pose error can be decomposed into position error and attitude error. The position error can be obtained by calculating the Euclidean distance or vector difference between the current end effector position vector and the target end effector position vector. The attitude error can be calculated in various ways. For example, when the pose is represented by quaternions, the attitude error quaternion can be obtained by calculating the product of the inverses of the current pose quaternion and the target pose quaternion, and then converted into a rotation vector or Euler angle form; or, when the pose is represented by a homogeneous transformation matrix, the pose error matrix can be obtained by calculating the product of the inverses of the current transformation matrix and the target transformation matrix, and then the position error vector and attitude error vector can be extracted from this matrix.

[0083] Based on this, calculating the desired velocity of the end effector in the task space based on the proportional gain matrix and the pose error means mapping the calculated pose error to the desired velocity in the task space through a proportional gain matrix. The proportional gain matrix is ​​a matrix that weights the pose error, and its diagonal elements can correspond to the proportional gain coefficients of each degree of freedom in the task space (e.g., position in the X, Y, and Z directions and attitude about the X, Y, and Z axes). These coefficients can be adjusted empirically or through system identification. Furthermore, the proportional gain matrix can also be designed as a non-diagonal matrix to handle more complex task or system dynamics, allowing coupled control between different degrees of freedom; or, considering that pose error typically includes position error and attitude error, the proportional gain matrix can be designed as a block matrix, with one sub-block handling position error and another sub-block handling attitude error, each corresponding to a different proportional gain.

[0084] This application effectively solves the problems of inaccurate error handling and unstable control response in traditional methods by clearly defining the calculation method of pose error and introducing a proportional gain matrix to transform the error into the desired velocity. First, accurate pose error calculation provides reliable input for subsequent velocity generation, avoiding accumulated errors or control deviations caused by fuzzy error estimation. Second, the application of the proportional gain matrix makes the velocity response of the robot's end effector smooth and controllable, significantly reducing oscillations or abrupt changes during motion, thus ensuring the continuity and stability of the robot's movements. This mechanism not only improves the accuracy of control but also enhances the system's adaptability and robustness to different task environments through an adjustable gain matrix. Building upon the basic scheme's generation of high-precision target pose through a diffusion strategy network, this scheme ensures a smooth transition from high-level decision-making to low-level execution, preventing the advantages of high-level decision-making from being negated by inaccurate control layer decisions. This enables the mobile robot to complete complex tasks more stably and accurately.

[0085] S5: Based on the desired speed, the current state of the robot, and system constraints, the base speed and the joint speed of the robotic arm are solved in real time to obtain the target speed command.

[0086] Please see Figure 7 , Figure 7 A specific implementation of step S5 is shown below: S51: Construct an operability-aware optimization control problem based on the desired speed, the robot's current state, and the system constraints. S52: Solve the optimization control problem in real time using a dimensionality-reduced quadratic programming solver to obtain the target speed command by calculating the base speed and the robot arm joint speed.

[0087] In some embodiments described above in this application, a method is proposed to solve the base velocity and robotic arm joint velocity in real time based on the desired velocity, the robot's current state, and system constraints, in order to generate target velocity commands to drive robot motion. However, in its implementation, because system constraints (such as joint velocity limits and collision avoidance) in dynamic and complex environments are not efficiently modeled and processed in real time, the solution process may suffer from excessive computational burden and significant delays, which in turn may lead to control command jitter, discontinuous trajectories, or violations of physical constraints, increasing operational risks and instability.

[0088] In response, this application further proposes a method for obtaining a target speed command by solving the base speed and the robot joint speed in real time based on the desired speed, the current state of the robot, and system constraints. Specifically, the method includes: constructing an operability-aware optimization control problem based on the desired speed, the current state of the robot, and the system constraints; and solving the base speed and the robot joint speed in real time using a dimensionality-reduced quadratic programming solver to obtain the target speed command.

[0089] Specifically, constructing an operability-aware optimization control problem aims to maximize or minimize a certain performance index by optimizing the robot's motion commands, while satisfying task expectations, current physical state, and various system constraints. "Operability-aware" means that the optimization process considers not only the execution of the current task but also the robot's future motion flexibility and ability to avoid singularities. In one implementation, objectives such as desired velocity tracking, system constraint satisfaction, and operability maintenance can be combined into a quadratic objective function with different weighting coefficients, and then combined with linear or nonlinear constraints to form a weighted least squares optimization problem. For example, the objective function may include an error term between desired and actual velocities, a regularization term for joint velocities, and an operability index term, while constraints may include upper and lower limits for joint velocities, joint angle limits, base velocity limits, and collision avoidance inequalities. In another implementation, the robot's dynamic model, desired trajectory, current state, and all system constraints can be incorporated into a rolling optimization framework within a finite time window. In each control cycle, an optimization problem is solved to obtain the optimal speed command at the current moment and to predict the robot's behavior over a period of time in the future, thereby achieving smooth tracking of the desired speed and maintaining operability while satisfying constraints.

[0090] Based on this, a dimensionality-reduced quadratic programming solver is used to solve the optimization control problem in real time for the base speed and the robot arm joint speed, thereby obtaining the target speed command. A dimensionality-reduced quadratic programming solver is an algorithm specifically designed to solve quadratic programming problems. Its characteristic is that it can effectively reduce the dimensionality or complexity of the problem through certain technical means, thereby significantly improving the solution speed and making it suitable for real-time control scenarios. In one implementation, a dimensionality-reducing solver based on the active set method can be used. This solver iteratively identifies and maintains a set of "active" constraints, decomposing the original problem into a series of unconstrained or equality-constrained subproblems. By adding or removing active constraints in each iteration, it gradually approaches the optimal solution. In another implementation, a dimensionality-reducing solver based on the interior-point method can be used. This method introduces a barrier function to integrate the constraints into the objective function, transforming the original problem into a series of unconstrained or equality-constrained problems, and solving them iteratively using methods such as Newton's method. Through preprocessing or variable substitution, the scale of the linear system to be solved in each iteration is reduced.

[0091] This application addresses the problem of control command jitter, discontinuous trajectories, or violations of physical constraints caused by inefficient modeling and real-time processing of system constraints in dynamic and complex environments. Specifically, by constructing an operability-aware optimization control problem, the desired speed, the robot's current state, and all system constraints (such as joint speed limits, joint angle limits, base speed limits, and collision avoidance inequality constraints based on geometric models) are integrated into a unified mathematical framework. This ensures that the generated base speed and robotic arm joint speed commands meet task requirements while strictly adhering to physical constraints and maintaining the robot's motion flexibility. Furthermore, a dimensionality-reduced quadratic programming solver is used to solve this optimization problem in real time, significantly reducing computational complexity and ensuring fast and stable output of target speed commands in dynamic environments, avoiding control jitter and trajectory interruptions caused by computational delays. This enables the mobile robot to perform motion tasks in a smoother, safer, and more reliable manner in complex and changing environments, thereby improving the overall system robustness and operational efficiency.

[0092] Furthermore, the system constraints include upper and lower limits for joint velocity, joint angle limits, base velocity limits, and collision avoidance inequality constraints based on the geometric model.

[0093] The joint speed upper and lower limits are used to define the maximum and minimum allowable movement speeds of each joint in the robotic arm. This prevents joints from experiencing mechanical wear, vibration, or instability due to excessive speed, thus ensuring the smooth operation and lifespan of the robotic arm. This can be achieved by setting a speed saturator in the underlying controller to truncate the desired joint speed; or by using an optimization algorithm in the motion planning layer to ensure that the generated joint speed trajectories always remain within a preset safe range.

[0094] Joint angle limits are used to define the minimum and maximum permissible rotational or translational angles of each joint in a robotic arm. This effectively prevents joint movement from exceeding its physical limits, preventing structural damage to the robotic arm or interference with itself or the environment. These limits can be implemented by defining the effective workspace of the joints in the robot's kinematic model, allowing configurations exceeding angle limits to be automatically avoided during inverse kinematics solving or trajectory generation; or by setting soft limits on the joint encoders or position sensors, triggering deceleration or stop commands when the joint approaches its physical limits.

[0095] The base speed limit is used to constrain the maximum linear and angular velocities of the mobile robot's base. This is crucial for the stable movement of the mobile robot in complex environments, preventing the base from slipping, tipping over, or generating excessive inertial forces due to excessive speed, which could affect the gripping or manipulating accuracy of the robotic arm. This limitation can be achieved by integrating a speed limiting module into the base motion controller to ensure that the speed commands sent to the drive motors do not exceed a preset safety threshold; or by dynamically adjusting the maximum permissible speed of the base based on environmental factors such as current terrain and load.

[0096] Collision avoidance inequality constraints based on geometric models are used to ensure that the robot does not collide with obstacles in the environment or with itself during movement. This constraint involves establishing a geometric model of the robot body and its working environment, and calculating the distances between each robot component and obstacles in real time. These distances are then transformed into a series of inequalities requiring that they always exceed a preset safe distance. Specific implementation methods can include: using simplified geometric models such as bounding boxes, spheres, or capsules to represent robot links and obstacles, and determining the existence of collision risk by calculating the minimum distance between these simplified models; or using more refined mesh models or point cloud data, combined with distance field technology (such as symbolic distance field SDF), to assess collision risk in real time and generate avoidance paths. S6: Send the target speed command to the robot's underlying actuator to drive the robot to complete the motion task and provide real-time feedback on the status to the perception and control module.

[0097] In one specific embodiment, a smart guidance and service robot for bank branches is provided. Its task is to actively identify customers who need assistance in a dynamic and crowded bank branch, guide them to a designated counter, and deliver business documents or smart devices (such as a signature board).

[0098] The following approach is used in the embodiments of this application: Multi-view perception: Through multiple cameras on the robot's head and chest, it can still keep track of customers from other perspectives even if they are temporarily blocked by others.

[0099] Diffusion strategy decision: Based on multi-view fusion features, it can stably predict customer movement trends and delivery points. Even if customers temporarily change direction or stop, it can still generate a smooth and reasonable approach path.

[0100] Operability-optimized control: When approaching the customer, the ReM-QP controller prevents the robotic arm from extending into unusual postures, ensuring a smooth and natural delivery process, while the base can be finely adjusted to maintain the optimal operating distance.

[0101] The embodiments of this application can improve the robustness of customer identification and tracking, and reduce task interruption rate. They enable smooth and safe movement in a human-machine collaborative environment, enhancing the service experience.

[0102] In another specific embodiment, an application scenario for intelligent management and delivery of sterile instruments in the operating room is provided. The task of this scenario is that, in the operating room environment, a mobile operating robot takes out a designated surgical instrument from the sterile instrument cabinet according to the instructions of the nurse or the system, and delivers it smoothly and accurately to the surgeon or scrub nurse next to the operating table.

[0103] The embodiments of this application can effectively suppress specular interference from a single viewpoint through multi-view fusion. Even if the instrument is partially obscured (e.g., by other instruments or people), its position can still be inferred through global representation. The smooth delivery trajectory demonstrated by the diffusion strategy learning expert generates a target pose sequence with temporal consistency, avoiding abrupt command changes from the source. ReM-QP strictly adheres to smaller speed and acceleration constraints during solution and incorporates operability optimization to ensure extremely smooth and controllable movements when approaching the doctor or operating table, avoiding any accidental collisions.

[0104] Please refer to Figure 8 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a hybrid control device for a mobile operating robot, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0105] like Figure 8As shown, the hybrid control device for the mobile robot in this embodiment includes: a data acquisition module 71, a feature extraction module 72, a target pose generation module 73, a desired velocity calculation module 74, a velocity command generation module 75, and a motion task driving module 76, wherein: The data acquisition module 71 is used to synchronously acquire RGB images from multiple perspectives through multiple cameras deployed on the robot, and synchronously acquire the robot's body state data. The feature extraction module 72 is used to perform convolutional feature extraction on the RGB images from each viewpoint to obtain feature maps from each viewpoint, and to perform feature fusion on the feature maps across viewpoints to generate a global scene representation. The target pose generation module 73 is used to encode the global scene representation and the current state of the robot into a conditional vector, and input it into a pre-trained diffusion policy network to generate the target pose of the end effector in the world coordinate system. The expected velocity calculation module 74 is used to calculate the expected velocity of the end effector in the task space based on the error between the current end effector pose and the target pose. The speed command generation module 75 is used to solve the base speed and the robot joint speed in real time based on the desired speed, the current state of the robot and system constraints, so as to obtain the target speed command. The motion task driving module 76 is used to send the target speed command to the robot's underlying actuator to drive the robot to complete the motion task and to provide real-time feedback on the status to the perception and control module.

[0106] Furthermore, the feature extraction module 72 includes: The feature map extraction unit is used to extract the high-dimensional semantic and geometric features of the RGB image from each viewpoint through a pre-trained convolutional neural network to obtain the feature map of each viewpoint. The feature stitching unit is used to flatten and stitch the feature maps from each viewpoint to generate stitched features; The weight calculation unit is used to calculate the correlation weight between the stitched features from each perspective through a multi-head self-attention mechanism, and to fuse the stitched features based on the correlation weight to obtain the weighted fused features. The feature aggregation unit is used to aggregate the weighted fused features to generate the global scene representation.

[0107] Furthermore, the target pose generation module 73 also includes the following: The data acquisition module is used to collect teleoperation demonstration data from multiple perspectives, wherein the teleoperation demonstration data includes multi-view RGB image sequences, robot state sequences, and end-effector trajectories. The noise addition module is used to add noise to the terminal trajectory and construct noise and denoised sample pairs; The model training module is used to train the diffusion strategy network to learn the denoising process of recovering the end trajectory from the noise based on the images in the multi-view RGB image sequence and the robot state in the robot state sequence, and to obtain the trained diffusion strategy network.

[0108] Furthermore, the desired speed calculation module 74 includes: The pose error calculation unit is used to calculate the pose error based on the current end pose and the target pose. A desired velocity generation unit is used to generate the desired velocity of the end effector in the task space based on the proportional gain matrix and the pose error.

[0109] Furthermore, the speed command generation module 75 includes: An optimization control problem construction unit is used to construct an operability-aware optimization control problem based on the desired speed, the current state of the robot, and the system constraints. The speed command generation unit is used to solve the optimization control problem in real time by using a dimensionality-reduced quadratic programming solver to obtain the target speed command by calculating the base speed and the joint speed of the robotic arm.

[0110] Furthermore, the data acquisition module 71 includes: An image acquisition unit is used to simultaneously acquire RGB images from multiple perspectives at the same time using multiple cameras deployed on the robot. The body state data acquisition unit is used to synchronously acquire the body state data of the robot through state sensors, wherein the body state data includes the base position and attitude, the joint state of the robotic arm, and the current pose of the end effector.

[0111] Furthermore, the system constraints include upper and lower limits for joint velocity, joint angle limits, base velocity limits, and collision avoidance inequality constraints based on the geometric model.

[0112] Traditional mobile robots often face challenges such as insufficient perception robustness, control jitter, and unstable decision-making when operating autonomously in unstructured environments. For example, in automated services and security patrols in the fintech sector, existing systems often rely on single-view cameras and predetermined paths, making them prone to losing targets in crowded areas or when visual obstruction occurs, leading to service interruptions. In the fields of medical assistance and rehabilitation care, existing robot systems often separate navigation and operation, resulting in slow adjustments and discontinuous trajectories when approaching beds or operating tables, and potentially causing medical risks due to control jitter. While the industry has attempted to introduce deep learning models to improve perception and decision-making capabilities, issues such as environmental interference and data distribution deviations frequently lead to sudden decision changes or discontinuous control. Furthermore, the lack of systematic modeling of physical constraints results in execution-level jitter, making it difficult to guarantee safety.

[0113] In response, this application provides a mobile operating robot. Please refer to the following for details. Figure 9 , Figure 9 This is a basic structural block diagram of the mobile operation robot in this embodiment. The mobile operation robot 8 includes a multi-channel camera 81, a robot body 82, a status sensor 83, and a controller 84.

[0114] The system includes a multi-camera 81 for synchronously acquiring RGB images of the environment from different perspectives; a robot body 82 including a movable chassis, a robotic arm, and an end effector; a state sensor 83 for acquiring the position and orientation of the chassis, the joint state of the robotic arm, and the pose of the end effector; and a controller 84 including a memory 841 and a processor 842. The memory 841 stores a computer program, and when the processor executes the computer program, it is used to implement a hybrid control method for the mobile robot.

[0115] In this embodiment, the multi-camera 81 is configured to simultaneously acquire RGB images of the environment from different perspectives. This ensures comprehensive scene information is obtained even under visual occlusion or dynamic interference, avoiding blind spots caused by a single perspective and thus enhancing the robustness of environmental understanding. The robot body 82 consists of a movable chassis, a robotic arm, and an end effector, supporting flexible movement and precise operation of the robot in space to adapt to changing task requirements. The state sensor 83 acquires the position and attitude of the chassis, the joint state of the robotic arm, and the pose of the end effector in real time, providing accurate internal state feedback for control decisions and reducing control jitter caused by state uncertainty. The controller 84 implements a hybrid control method for the mobile robot by executing a stored computer program. This method generates coordinated control commands based on multi-view perception and state data, achieving deep fusion of perception and control, ensuring continuous and stable decision-making processes, and avoiding motion interruptions or safety risks.

[0116] Through the above technical solution, this application effectively solves the problems of inaccurate perception, unstable control, and discontinuous decision-making in unstructured environments for the mobile operating robot 8. Specifically, the combination of multi-view synchronous perception by the multi-camera 81 and real-time status monitoring by the status sensor 83 provides comprehensive environmental information and accurate robot status, enabling the controller 84 to generate smooth and continuous control commands, significantly improving the robot's operational stability and safety in dynamic and complex scenarios. Therefore, this mobile operating robot 8 can achieve stable, continuous, and safe autonomous operation in scenarios with high reliability requirements, such as financial security and medical assistance.

[0117] The memory 841 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 841 may be an internal storage unit of the mobile operating robot 87, such as the hard disk or memory of the mobile operating robot 87. In other embodiments, the memory 841 may also be an external storage device of the mobile operating robot 87, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the mobile operating robot 87. Of course, the memory 841 may also include both internal storage units and external storage devices of the mobile operating robot 87. In this embodiment, the memory 841 is typically used to store the operating system and various application software installed on the mobile operating robot 87, such as the program code of the hybrid control method of the mobile operating robot. In addition, the memory 841 can also be used to temporarily store various types of data that have been output or will be output.

[0118] In some embodiments, processor 842 may be a central processing unit (CPU), controller 84, microcontroller 84, microprocessor, or other data processing chip. This processor 842 is typically used to control the overall operation of the mobile manipulator robot 87. In this embodiment, processor 842 is used to run program code stored in memory 841 or process data, for example, to run the program code of the hybrid control method for the mobile manipulator robot 8 described above, to implement various embodiments of the hybrid control method for the mobile manipulator robot.

[0119] This application also provides another embodiment, namely, a computer-readable storage medium storing a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the above-described hybrid control method for a mobile operating robot.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0121] Obviously, the embodiments described above are merely some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the scope of this application. This application can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of protection of this application.

Claims

1. A hybrid control method for a mobile operating robot, characterized in that, include: Multi-view RGB images are simultaneously acquired by multiple cameras deployed on the robot, and the robot's body state data is also acquired simultaneously. Convolutional feature extraction is performed on the RGB images from each viewpoint to obtain feature maps for each viewpoint, and feature fusion is performed on the feature maps across viewpoints to generate a global scene representation. The global scene representation and the robot's current state are encoded into a conditional vector and input into a pre-trained diffusion policy network to generate the target pose of the end effector in the world coordinate system. The expected velocity of the end effector in the task space is calculated based on the error between the current end effector pose and the target pose. Based on the desired speed, the current state of the robot, and system constraints, the base speed and the joint speed of the robotic arm are solved in real time to obtain the target speed command; The target speed command is sent to the robot's underlying actuator to drive the robot to complete the motion task and to provide real-time feedback on the status to the perception and control module.

2. The hybrid control method for a mobile operating robot according to claim 1, characterized in that, The process involves performing convolutional feature extraction on the RGB images from each viewpoint to obtain feature maps for each viewpoint, and then fusing the feature maps across viewpoints to generate a global scene representation, including: High-dimensional semantic and geometric features of the RGB images from each viewpoint are extracted using a pre-trained convolutional neural network to obtain the feature maps for each viewpoint. The feature maps from each perspective are flattened and stitched together to generate stitched features; The correlation weights between the stitched features from each viewpoint are calculated using a multi-head self-attention mechanism, and the stitched features are fused based on the correlation weights to obtain weighted fused features. The weighted fused features are aggregated to generate the global scene representation.

3. The hybrid control method for a mobile operating robot according to claim 1, characterized in that, Before encoding the global scene representation and the robot's current state into a conditional vector and inputting it into a pre-trained diffusion policy network to generate the target pose of the end effector in the world coordinate system, the method further includes: Collect multi-view teleoperation demonstration data, wherein the teleoperation demonstration data includes multi-view RGB image sequences, robot state sequences, and end-effector trajectories; Noise is added to the terminal trajectory to construct noise-denoised sample pairs; Using the images in the multi-view RGB image sequence and the robot state in the robot state sequence as conditions, the diffusion strategy network is trained based on the noise and denoised samples to learn the denoising process of recovering the end trajectory from the noise, thus obtaining the trained diffusion strategy network.

4. The hybrid control method for a mobile operating robot according to claim 1, characterized in that, The step of calculating the expected velocity of the end effector in the task space based on the error between the current end effector pose and the target pose includes: Calculate the pose error based on the current end pose and the target pose; The desired speed of the end effector in the task space is based on the proportional gain matrix and the pose error.

5. The hybrid control method for a mobile operating robot according to claim 1, characterized in that, The process of obtaining the target speed command by real-time solving of the base speed and the robotic arm joint speed based on the desired speed, the robot's current state, and system constraints includes: An operability-aware optimization control problem is constructed based on the desired speed, the robot's current state, and the system constraints. The optimization control problem is solved in real time by using a dimensionality-reduced quadratic programming solver to obtain the target speed command by solving the base speed and the robot arm joint speed.

6. The hybrid control method for a mobile operating robot according to any one of claims 1 to 5, characterized in that, The process of simultaneously acquiring multi-view RGB images through multiple cameras deployed on the robot, and simultaneously obtaining the robot's body state data, includes: The RGB images are simultaneously acquired from multiple perspectives by multiple cameras deployed on the robot at the same time. The robot's body state data is acquired synchronously through state sensors. The body state data includes the base position and posture, the joint state of the robotic arm, and the current posture of the end effector.

7. The hybrid control method for a mobile operating robot according to any one of claims 1 to 5, characterized in that, The system constraints include upper and lower limits for joint velocity, joint angle limits, base velocity limits, and collision avoidance inequalities based on the geometric model.

8. A hybrid control device for a mobile operating robot, characterized in that, include: The data acquisition module is used to simultaneously acquire RGB images from multiple perspectives using multiple cameras deployed on the robot, and to simultaneously acquire the robot's body state data. The feature extraction module is used to perform convolutional feature extraction on the RGB images from each viewpoint to obtain feature maps for each viewpoint, and to perform feature fusion on the feature maps across viewpoints to generate a global scene representation. The target pose generation module is used to encode the global scene representation and the robot's current state into a conditional vector, and input it into a pre-trained diffusion policy network to generate the target pose of the end effector in the world coordinate system. The expected velocity calculation module is used to calculate the expected velocity of the end effector in the task space based on the error between the current end effector pose and the target pose. The speed command generation module is used to solve the base speed and the robot joint speed in real time based on the desired speed, the current state of the robot and system constraints, so as to obtain the target speed command. The motion task driving module is used to send the target speed command to the robot's underlying actuator to drive the robot to complete the motion task and to provide real-time feedback on the status to the perception and control module.

9. A mobile operating robot, characterized in that, include: Multiple cameras are used to simultaneously capture RGB images of the environment from different perspectives; The robot body includes a movable chassis, a robotic arm, and an end effector; A status sensor is used to acquire the position and orientation of the chassis, the joint state of the robotic arm, and the pose of the end effector. A controller, comprising a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the hybrid control method for a mobile operating robot as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the hybrid control method for a mobile operating robot as described in any one of claims 1 to 7.

Images