Multi-modal sensing humanoid robot action self-adaptive control method and multi-modal sensing humanoid robot action self-adaptive control system

Abstract

The invention discloses a multi-modal perception humanoid robot action adaptive control method and system, and belongs to the technical field of computers, and the method comprises the steps: obtaining first multi-modal data; the first multi-modal data comprises first image data, first point cloud data and first force sense data; performing data fusion on the first image data and the first point cloud data, and constructing a three-dimensional model of the scene in real time; generating an optimal path according to the real-time environment data of the three-dimensional model, converting the optimal path into a joint instruction of the humanoid robot, and generating an action control strategy; according to the action control strategy, arm movement of the robot is adjusted through visual servo control, and first operation data are obtained; the path and action of the robot are adjusted in a closed-loop control mode, and second operation data are obtained; and reinforcement learning and regular evaluation are carried out through a DDPG algorithm, and an action control strategy is continuously optimized. According to the invention, multi-modal sensing robot motion adaptive control with strong adaptive capability, high operation precision and good robustness can be realized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and system for adaptive motion control of a humanoid robot with multi-modal perception. Background Art

[0002] Currently in the field of intelligent manufacturing, humanoid robots are gradually being used for complex tasks such as automated assembly, precision welding, quality inspection, etc. However, most traditional robots rely on predefined motion trajectories and operating procedures, and have many limitations when facing changing production environments and unstructured tasks. These systems usually require a lot of manual programming, are difficult to effectively respond to real-time production scenarios, and lack flexibility and adaptability.

[0003] In recent years, video-based imitation learning technology has gradually become an important development direction. This technology allows robots to observe human demonstration videos, learn and imitate corresponding actions, thereby reducing programming costs and improving the robot's ability to perform complex tasks. Although video-based imitation learning robot technology has improved the flexibility of industrial operations to a certain extent, it still has many shortcomings in practical applications, especially when faced with high-precision and complex operation tasks, exposing the following technical bottlenecks:

[0004] Lack of adaptive capabilities and limited ability to cope with dynamic environments: Current video imitation learning methods usually generate operation plans based on pre-trained offline models. When the actual production environment changes (such as workpiece position offset, shape change, or task requirements adjustment), the robot lacks the ability to adjust in real time. This makes the robot prone to operational errors or task failures in dynamic production scenarios, thus affecting production efficiency and quality.

[0005] Limited operation accuracy and lack of error compensation mechanism: Traditional robot systems based on imitation learning are usually unable to perceive and adjust errors in real time during operation. Once the robot encounters position deviation or operation error during operation, it is often difficult to automatically correct it through the existing system, resulting in a high task failure rate. This lack of error compensation capability is particularly evident in tasks such as precision assembly, grinding, and welding, which seriously restricts the application of robots in high-precision industrial operations.

[0006] Lack of collaborative perception: Some existing technologies usually rely solely on video information or depth images to make operational decisions. This reliance on a single data source makes the system less robust in complex scenarios and makes it difficult to achieve comprehensive perception and understanding of the environment. Summary of the invention

[0007] In order to solve at least one of the above problems, the embodiments of the present application propose a method and system for adaptive control of humanoid robot motion with multi-modal perception, strong adaptability, high operating accuracy and good robustness.

[0008] To achieve the above-mentioned purpose, an embodiment of the present application provides a method for adaptively controlling the motion of a humanoid robot with multimodal perception, the method comprising:

[0009] The data in the environment is collected simultaneously by an RGB-D camera, a laser radar and a force sensor and preprocessed to obtain first multimodal data; wherein the first multimodal data includes first image data, first point cloud data and first force data;

[0010] Using a deep learning algorithm to perform data fusion on the first image data and the first point cloud data, and constructing a three-dimensional model of the operation scene in real time according to the result of the data fusion, and obtaining real-time environmental data of the three-dimensional model;

[0011] Generate an optimal path according to the real-time environmental data, convert the optimal path into joint instructions of the humanoid robot through an inverse dynamics model, and generate a motion control strategy;

[0012] According to the motion control strategy, adjusting the arm movement of the humanoid robot through visual servo control so that the humanoid robot performs the task and obtains first operation data;

[0013] During the operation of the humanoid robot, according to the first force sense data, a path and a movement of the humanoid robot are adjusted by a closed-loop control method to obtain second operation data;

[0014] During the operation of the humanoid robot, the motion control strategy is reinforced learned and regularly evaluated by using the DDPG algorithm in combination with the first operation data and the second operation data, so as to continuously optimize the motion control strategy.

[0015] In some embodiments, the method of simultaneously collecting data in the environment through an RGB-D camera, a laser radar, and a force sensor and preprocessing the data to obtain the first multimodal data includes the following steps:

[0016] The RGB-D camera collects the color image and depth information in the environment to obtain an RGB image;

[0017] Acquire three-dimensional point cloud data in the environment by using a laser radar;

[0018] During the working process of the humanoid robot, the contact force of the humanoid robot on the target workpiece is collected through the force sensor to obtain force sensor data;

[0019] Performing gamma correction and white balance adjustment on the RGB image to obtain depth image data;

[0020] Performing bilateral filtering on the first processed image to obtain first image data;

[0021] Downsampling the three-dimensional point cloud data to obtain key point cloud data;

[0022] Using a random sampling consistency algorithm to remove outliers and noise data in the key point cloud data to obtain first point cloud data;

[0023] Kalman filtering is performed on the force sensor data to obtain first force data.

[0024] In some embodiments, the step of fusing the first image data and the first point cloud data using a deep learning algorithm, constructing a three-dimensional model of the operation scene in real time according to the result of the data fusion, and acquiring real-time environmental data of the three-dimensional model includes the following steps:

[0025] Using a three-dimensional convolutional neural network to extract three-dimensional features of the first image data and the first point cloud data;

[0026] According to the three-dimensional features, performing data fusion processing on the first image data and the first point cloud data to obtain first fused data;

[0027] According to the first fusion data, real-time positioning of the humanoid robot is performed in a dynamic environment, and a three-dimensional model is constructed; wherein the three-dimensional model includes a three-dimensional map and an object model;

[0028] Converting the first point cloud data into a voxel map;

[0029] continuously updating the three-dimensional model according to the voxel map;

[0030] According to the three-dimensional model, real-time environmental data of the humanoid robot during its working process is obtained.

[0031] In some embodiments, generating an optimal path according to the real-time environmental data, converting the optimal path into joint instructions of the humanoid robot through an inverse dynamics model, and generating a motion control strategy include the following steps:

[0032] Using an RRT algorithm, based on the three-dimensional environment map and the positioning of the end effector of the humanoid robot, an optimal path of the humanoid robot from a starting position to a target position is generated;

[0033] Discretize the optimal path to obtain position information of each path point of the end effector on the optimal path;

[0034] According to the position information, solving a first joint angle and a first joint velocity of the end effector at the path point by inverse dynamics;

[0035] According to the joint angle and the joint speed, a path is smoothed by a trajectory generation algorithm, and a second speed and a second acceleration between each of the path points are calculated;

[0036] generating a joint instruction according to the position information, the first joint angle, the first joint velocity, the second velocity and the second acceleration;

[0037] A motion control strategy is generated according to the joint instructions.

[0038] In some embodiments, during the operation of the humanoid robot, adjusting the path and action of the humanoid robot by a closed-loop control method according to the first force data to obtain second operation data includes the following steps:

[0039] According to the first force sense data, the humanoid robot detects the contact force between the humanoid robot and the target workpiece to obtain a contact force monitoring result;

[0040] Predicting the future behavior of the humanoid robot according to the contact force monitoring result by a model predictive control algorithm to obtain a behavior prediction result;

[0041] Adaptively adjusting the path and action of the humanoid robot according to the behavior prediction result;

[0042] The humanoid robot is controlled to perform an operation according to the adjusted path and action to obtain second operation data.

[0043] In some embodiments, during the operation of the humanoid robot, the motion control strategy is reinforced learned and periodically evaluated by combining the first operation data and the second operation data through the DDPG algorithm to continuously optimize the motion control strategy, including the following steps:

[0044] Generate a preliminary policy model for parameter sharing by imitating historical operation videos and demonstration data through behavioral cloning technology;

[0045] Based on a deep deterministic policy gradient algorithm, jointly train the preliminary policy model in multiple scenarios, and optimize the policy of the preliminary policy model according to the first operation data and the second operation data to obtain a second policy model;

[0046] Performing an operation performance evaluation according to the task completion degree of the humanoid robot during the working process to obtain an operation performance evaluation result;

[0047] The second strategy model is adaptively adjusted according to the operation performance evaluation result to achieve continuous optimization of the action control model.

[0048] In some embodiments, the method further comprises the following steps:

[0049] Uploading the task execution data obtained by the humanoid robot performing the operation task to the cloud platform for storage; wherein the task execution data includes the robot task log, the real-time environment data, the first multimodal data, and the action execution strategy;

[0050] According to the task execution data stored in the cloud platform, performing, during the operation of the humanoid robot, a step of performing reinforcement learning and regular evaluation on the motion control strategy by combining the first operation data and the second operation data through a DDPG algorithm to continuously optimize the motion control strategy;

[0051] Performing big data analysis on the task execution data stored on the cloud platform, and optimizing the motion control strategy according to the result of the big data analysis;

[0052] The optimized motion control strategy model is synchronized to the humanoid robot.

[0053] To achieve the above purpose, another aspect of the embodiment of the present application provides a multi-modal perception humanoid robot motion adaptive control system, the system comprising:

[0054] A multimodal perception module, used to simultaneously collect data in the environment through an RGB-D camera, a laser radar, and a force sensor and perform preprocessing to obtain first multimodal data; wherein the first multimodal data includes first image data, first point cloud data, and first force data;

[0055] A data fusion and environment modeling module, configured to perform data fusion on the first image data and the first point cloud data using a deep learning algorithm, and to construct a three-dimensional model of the operation scene in real time according to the result of the data fusion, and to obtain real-time environment data of the three-dimensional model;

[0056] A path planning and control module, used to generate an optimal path according to the real-time environmental data, convert the optimal path into joint instructions of the humanoid robot through an inverse dynamics model, and generate a motion control strategy; and also used to adjust the arm movement of the humanoid robot through visual servo control according to the motion control strategy, so that the humanoid robot performs the task and obtains first operation data;

[0057] an adaptive task execution and force feedback module, configured to adjust the path and action of the humanoid robot in a closed-loop control manner according to the first force data during the operation of the humanoid robot, so as to obtain second operation data;

[0058] The reinforcement learning and strategy optimization module is used to perform reinforcement learning and regular evaluation of the motion control strategy in combination with the first operation data and the second operation data through the DDPG algorithm during the operation of the humanoid robot, so as to continuously optimize the motion control strategy.

[0059] In some embodiments, the system further comprises:

[0060] A cloud data management and remote collaboration module is used to upload the task execution data obtained by the humanoid robot when performing the operation task to the cloud platform for storage; wherein the task execution data includes the robot task log, the real-time environment data, the first multimodal data, and the action execution strategy; it is also used to execute the steps of continuously optimizing the action control strategy by using the DDPG algorithm to perform reinforcement learning and regular evaluation on the action control strategy in combination with the first operation data and the second operation data during the operation of the humanoid robot according to the task execution data stored on the cloud platform; it is also used to perform big data analysis based on the task execution data stored on the cloud platform, and optimize the action control strategy according to the results of the big data analysis; it is also used to synchronize the optimized action control strategy model to the humanoid robot.

[0061] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0062] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0063] The embodiments of the present application include at least the following beneficial effects: The present application provides a method and system for adaptive motion control of a humanoid robot with multimodal perception. The scheme acquires first multimodal data, and fuses the image data and the first point cloud data in the first multimodal data, thereby constructing a three-dimensional model of the operation scene in real time, and acquiring the real-time environmental data of the three-dimensional model, so as to obtain high-precision scene information. Through the deep learning algorithm, multimodal data fusion processing is performed to achieve a comprehensive understanding and dynamic adjustment of the operation scene. At the same time, reinforcement learning and inverse dynamics control strategies are introduced. Through closed-loop control technology, the humanoid robot can adapt to the operation path when performing complex tasks, ensuring high-precision, high-efficiency and high-adaptability production operations. Moreover, by optimizing the motion control strategy, the humanoid robot can show greater flexibility and robustness when facing variable production tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0065] Figure 1 is a step diagram of a multi-modal perception humanoid robot motion adaptive control method provided in an embodiment of the present application;

[0066] Figure 2 It is a flow chart of an embodiment provided by the embodiments of the present application;

[0067] Figure 3 It is a module schematic diagram of a multi-modal perception humanoid robot motion adaptive control system provided in an embodiment of the present application;

[0068] Figure 4 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0070] Although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0071] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0072] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0073] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0074] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0075] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0076] Robot: Unless otherwise specified, the robots mentioned in the embodiments of the present application are all humanoid robots.

[0077] Federated learning: It ensures data privacy by distributing the training models of multiple devices. It does not require data sharing and is suitable for scenarios with high data privacy requirements.

[0078] Big data: refers to large-scale data that exceeds traditional processing capabilities, and valuable information is extracted through big data technology.

[0079] Machine learning: Allowing computers to automatically learn from data and make predictions or decisions through supervised or unsupervised learning algorithms.

[0080] Deep learning: A machine learning method based on deep neural networks that can automatically extract features from complex data.

[0081] Reinforcement learning: A machine learning approach in which an agent interacts with its environment and adjusts its behavior based on feedback to maximize long-term rewards.

[0082] Reinforcement learning agent: In reinforcement learning, an agent that adjusts its behavior by interacting with its environment and receiving rewards.

[0083] Cloud platform: provides computing, storage and other resources through the Internet, which users can use on demand without the need for local hardware support.

[0084] Sensor: A device used to sense physical or chemical phenomena and convert them into electrical signals, such as temperature sensors, image sensors, etc.

[0085] Image recognition: Identifying objects or scenes in images through deep learning.

[0086] Cloud-based data aggregation: Data from different devices are aggregated to the cloud platform for integration and analysis to support decision optimization.

[0087] Task Scheduling: Arrange task execution according to task priority, time and other factors to improve efficiency and resource utilization.

[0088] Optimization algorithm: Find the optimal solution to a problem through iterative calculation.

[0089] Among the related technologies, although the current robot technology based on video imitation learning has improved the flexibility of industrial operations to a certain extent, it still has many shortcomings in practical applications, especially when faced with high-precision and complex operation tasks, it exposes the following technical bottlenecks:

[0090] Insufficient environmental perception capabilities, making it difficult to meet high-precision requirements: Existing video imitation learning systems mainly rely on RGB-D cameras for object detection and operation path planning. However, RGB-D cameras are easily disturbed by lighting changes, complex backgrounds, and object occlusion, resulting in reduced positioning and recognition accuracy. This is difficult to meet actual production requirements for assembly, welding, and quality inspection tasks that require micron-level precision. Therefore, existing systems lack the ability to perceive the position and shape of workpieces in the environment with high precision, making it difficult to ensure the accuracy of robot operations.

[0091] Lack of adaptive capabilities and limited ability to cope with dynamic environments: Current video imitation learning methods usually generate operation plans based on pre-trained offline models. When the actual production environment changes (such as workpiece position offset, shape change, or task requirements adjustment), the robot lacks the ability to adjust in real time. This makes the robot prone to operational errors or task failures in dynamic production scenarios, thus affecting production efficiency and quality.

[0092] Limited operation accuracy and lack of error compensation mechanism: Traditional robot systems based on imitation learning are usually unable to perceive and adjust errors in real time during operation. Once the robot encounters position deviation or operation error during operation, it is often difficult to automatically correct it through the existing system, resulting in a high task failure rate. This lack of error compensation capability is particularly evident in tasks such as precision assembly, grinding, and welding, which seriously restricts the application of robots in high-precision industrial operations.

[0093] Difficulty integrating multimodal data for collaborative perception: Existing technologies usually rely solely on video information or depth images for operational decisions, and fail to fully utilize the high-precision data provided by multiple sensors such as LiDAR and force sensors. This reliance on a single data source results in insufficient robustness of the system in complex scenarios, making it difficult to achieve comprehensive perception and understanding of the environment.

[0094] In view of this, the embodiment of the present application provides a multimodal perception humanoid robot motion adaptive control method and system, which combines video imitation learning, laser radar (LiDAR) perception, deep reinforcement learning (DRL) and closed-loop control technology to achieve precise operation and task adaptive adjustment of the robot in a dynamic industrial environment. The core of the embodiment of the present application is to obtain high-precision three-dimensional point cloud data and video information through a multimodal perception module, and use a deep learning algorithm to perform fusion processing of multi-source data, so as to achieve a comprehensive understanding and dynamic adjustment of the operation scene. At the same time, the introduction of reinforcement learning and inverse dynamics control strategies enables the robot to adaptively adjust the operation path when performing complex tasks, ensuring high-precision and high-efficiency production operations. Through the closed-loop control mechanism, the robot can detect operation deviations in real time during the execution process and automatically correct them, thereby significantly improving the stability and success rate of the operation. The robot motion adaptive control method of the embodiment of the present application not only has the ability to identify and locate objects in the environment with high precision, but also can continuously optimize the operation strategy through the reinforcement learning model, so that the humanoid robot can show stronger flexibility and robustness when facing variable production tasks.

[0095] The multi-modal perception humanoid robot motion adaptive control method provided in the embodiment of the present application relates to the field of computer technology. The multi-modal perception humanoid robot motion adaptive control method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in a blockchain network; the software can be an application of a multi-modal perception humanoid robot motion adaptive control method, etc., but is not limited to the above forms.

[0096] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0097] Figure 1 is an optional flow chart of the method for adaptively controlling the motion of a multi-modal-perceived humanoid robot provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S600.

[0098] Step S100, collecting data in the environment simultaneously through an RGB-D camera, a laser radar, and a force sensor and performing preprocessing to obtain first multimodal data; wherein the first multimodal data includes first image data, first point cloud data, and first force data;

[0099] Step S200, using a deep learning algorithm to perform data fusion on the first image data and the first point cloud data, and constructing a three-dimensional model of the operation scene in real time according to the result of the data fusion, and obtaining real-time environmental data of the three-dimensional model;

[0100] Step S300, generating an optimal path according to the real-time environmental data, converting the optimal path into joint instructions of the humanoid robot through an inverse dynamics model, and generating a motion control strategy;

[0101] Step S400, adjusting the arm movement of the humanoid robot through visual servo control according to the motion control strategy, so that the humanoid robot performs the task and obtains first operation data;

[0102] Step S500, during the operation of the humanoid robot, adjusting the path and movement of the humanoid robot by a closed-loop control method according to the first force sense data to obtain second operation data;

[0103] Step S600, during the operation of the humanoid robot, the motion control strategy is subjected to reinforcement learning and regular evaluation in combination with the first operation data and the second operation data through the DDPG algorithm to continuously optimize the motion control strategy.

[0104] In the steps S100 to S600 shown in the embodiment of the present application, by acquiring the first multimodal data, and fusing the image data and the first point cloud data in the first multimodal data, a three-dimensional model of the operation scene is constructed in real time, and the real-time environmental data of the three-dimensional model is acquired, high-precision scene information can be obtained. Multimodal data fusion processing is performed through a deep learning algorithm to achieve a comprehensive understanding and dynamic adjustment of the operation scene. At the same time, reinforcement learning and inverse dynamics control strategies are introduced. Through closed-loop control technology, the humanoid robot can adapt to the operation path when performing complex tasks, ensuring high-precision, high-efficiency and high-adaptability production operations. Moreover, by optimizing the motion control strategy, the humanoid robot can show greater flexibility and robustness when facing variable production tasks.

[0105] In some embodiments, step S100 may include but is not limited to the following steps S110 to S190:

[0106] Step S110, collecting color images and depth information in the environment through an RGB-D camera to obtain an RGB image;

[0107] Step S120, acquiring three-dimensional point cloud data in the environment through a laser radar;

[0108] Step S130, during the working process of the humanoid robot, collecting the contact force of the humanoid robot on the target workpiece through a force sensor to obtain force sensor data;

[0109] Step S140, performing gamma correction and white balance adjustment on the RGB image to obtain depth image data;

[0110] Step S150, performing bilateral filtering on the first processed image to obtain first image data;

[0111] Step S160, downsampling the three-dimensional point cloud data to obtain key point cloud data;

[0112] Step S170, using a random sampling consistency algorithm to remove outliers and noise data in the key point cloud data to obtain first point cloud data;

[0113] Step S190: Perform Kalman filtering on the force sensor data to obtain first force data.

[0114] In some embodiments, step S200 may include but is not limited to the following steps S210 to S260:

[0115] Step S210, extracting three-dimensional features of the first image data and the first point cloud data using a three-dimensional convolutional neural network;

[0116] Step S220, performing data fusion processing on the first image data and the first point cloud data according to the three-dimensional features to obtain first fused data;

[0117] Step S230, positioning the humanoid robot in real time in a dynamic environment according to the first fusion data, and constructing a three-dimensional model; wherein the three-dimensional model includes a three-dimensional map and an object model;

[0118] Step S240, converting the first point cloud data into a voxel map;

[0119] Step S250, continuously updating the three-dimensional model according to the voxel map;

[0120] Step S260: acquiring real-time environmental data of the humanoid robot during its operation according to the three-dimensional model.

[0121] In some embodiments, step S300 includes but is not limited to the following steps S310 to S360:

[0122] Step S310, using the RRT algorithm to generate an optimal path for the humanoid robot from a starting position to a target position based on the three-dimensional environment map and the end effector positioning of the humanoid robot;

[0123] Step S320, discretizing the optimal path to obtain position information of each path point of the end effector on the optimal path;

[0124] Step S330, solving a first joint angle and a first joint velocity of the end effector at the path point by inverse dynamics according to the position information;

[0125] Step S340, smoothing the path by a trajectory generation algorithm according to the joint angle and the joint velocity, and calculating a second velocity and a second acceleration between each of the path points;

[0126] Step S350, generating a joint instruction according to the position information, the first joint angle, the first joint speed, the second speed and the second acceleration;

[0127] Step S360: generating a motion control strategy according to the joint instructions.

[0128] In some embodiments, step S500 includes but is not limited to the following steps S510 to S540:

[0129] Step S510, the humanoid robot detects the contact force between the humanoid robot and the target workpiece according to the first force sense data to obtain a contact force monitoring result;

[0130] Step S520, predicting the future behavior of the humanoid robot according to the contact force monitoring result through a model predictive control algorithm to obtain a behavior prediction result;

[0131] Step S530, adaptively adjusting the path and action of the humanoid robot according to the behavior prediction result;

[0132] Step S540: Control the humanoid robot to perform an operation according to the adjusted path and action to obtain second operation data.

[0133] In some embodiments, step S600 includes but is not limited to the following steps S610 to S640:

[0134] Step S610, using behavior cloning technology to imitate historical operation videos and demonstration data to generate a preliminary strategy model for parameter sharing;

[0135] Step S620, based on a deep deterministic policy gradient algorithm, jointly train the preliminary policy model in multiple scenarios, optimize the policy of the preliminary policy model according to the first operation data and the second operation data, and obtain a second policy model;

[0136] Step S630, performing an operation performance evaluation according to the task completion degree of the humanoid robot during the working process to obtain an operation performance evaluation result;

[0137] Step S640: adaptively adjust the second strategy model according to the operation performance evaluation result to achieve continuous optimization of the action control model.

[0138] In some embodiments, the present application embodiment further includes the following steps S710 to S740:

[0139] Step S710, uploading the task execution data obtained by the humanoid robot performing the operation task to the cloud platform for storage; wherein the task execution data includes the robot task log, the real-time environment data, the first multimodal data, and the action execution strategy;

[0140] Step S720, performing, according to the task execution data stored in the cloud platform, a step of continuously optimizing the motion control strategy by performing reinforcement learning and regular evaluation on the motion control strategy by combining the first operation data and the second operation data through a DDPG algorithm during the operation of the humanoid robot;

[0141] Step S730, performing big data analysis on the task execution data stored in the cloud platform, and optimizing the motion control strategy according to the result of the big data analysis;

[0142] Step S740: synchronizing the optimized motion control strategy model to the humanoid robot.

[0143] Below, in conjunction with a specific scenario application example of a humanoid robot performing a task operation, the solution of the embodiment of the present application is described in detail and explained:

[0144] The embodiment of the present application provides a robot adaptive motion control method based on multi-modal high-precision perception, which is used to adaptively adjust its own motion control strategy according to the environment, so that the robot has flexible task execution capabilities. Figure 2 As shown, it can be implemented as the following steps:

[0145] S1. Multimodal data collection and preprocessing. After the robot is started, it collects multi-source data in the environment through RGB-D camera, LiDAR and force sensor at the same time. The collected multi-source data is denoised and filtered to provide high-quality input for subsequent processing.

[0146] Specifically, a high-resolution RGB-D camera such as Intel RealSense D435i or ZED 2 can be used to collect color images and depth information to capture the operating scene. First, gamma correction and white balance adjustment are performed on the acquired RGB image to reduce the impact of ambient lighting changes on image quality and ensure the accuracy of object detection. Next, bilateral filtering is used to process the depth image data to reduce the noise in the depth information, improve the clarity and accuracy of the edges, and obtain the first image data. The preprocessed first image data will be used for object recognition and three-dimensional reconstruction to ensure that the perceived environmental information has higher reliability.

[0147] In some embodiments, Ouster OS1 or Velodyne Alpha Prime laser radar can be used to obtain high-precision 3D point cloud data, and the obtained 3D point cloud data is preprocessed as follows: first, the point cloud data is downsampled by VoxelGrid Downsampling to retain key geometric information; then, the RANSAC (random sampling consistency) algorithm is used to remove outliers and noise data in the 3D point cloud data. This is conducive to reducing the computational burden and ensuring the quality and stability of the point cloud data.

[0148] In some embodiments, the contact force with the workpiece is monitored in real time by a force sensor. A six-dimensional force sensor may be used to collect contact force data. The data collected by the sensor is processed by Kalman filtering to eliminate noise and environmental interference, thereby obtaining a smoother and more accurate force signal as the first force data.

[0149] S2. Multimodal data fusion and environment modeling. The pre-processed RGB-D image (first image data) and LIDAR point cloud data (first point cloud data) are PointFusion fused to generate a three-dimensional map and object model of the environment to obtain three-dimensional map data. Through SLAM (Simultaneous Localization and Mapping) technology, a three-dimensional model of the operation scene is constructed in real time to ensure the positioning and navigation of the robot in a dynamic environment.

[0150] Specifically, through timestamp synchronization, multi-source data is aligned to ensure data consistency within the same frame. PointFusion technology is used to combine RGB-D data and point cloud features, and the fused three-dimensional features are extracted through a 3D convolutional neural network to improve the accuracy of object detection and environmental modeling. SLAM (Simultaneous Localization and Mapping) technology is further used to achieve real-time positioning and three-dimensional map construction of the robot in a dynamic environment to obtain three-dimensional map data. The point cloud data is converted into a voxel map so that the model of the three-dimensional map data can be continuously updated in dynamic scenes.

[0151] In some embodiments, complex task instructions can also be parsed by integrating a visual-language model (qwen2-72b), so that the robot has higher flexibility and execution efficiency when dealing with diverse tasks.

[0152] S3, path planning and motion control. Based on real-time environmental data, the RRT algorithm is used to generate the optimal obstacle avoidance path, and the optimal path is converted into robot joint instructions through the inverse dynamics model. The visual servo control module uses Dynamic Window Approach (DWA) to adjust the arm movement of the humanoid robot to ensure accurate alignment with the target object.

[0153] Path planning and motion control are responsible for ensuring that the robot can achieve high-precision operation in complex and changing industrial environments. Through path planning, visual servo control, force feedback and closed-loop control, the robot has the ability to flexibly adjust the operation path and adaptively control the motion, thus meeting the requirements of high precision and high efficiency in intelligent production.

[0154] Specifically, first, the RRT (Rapidly-exploring Random Tree Star) algorithm is used for path planning to ensure that the robot can avoid obstacles in complex industrial environments and reach the target location efficiently. Furthermore, the optimal path is constructed by sampling points in the configuration space. After the path is generated, the inverse dynamics model is used to convert the path into joint instructions of the robot arm. By adjusting the angles and speeds of each joint of the robot, it is ensured that the task can be accurately performed in actual operation.

[0155] In the embodiment of the present application, path planning and motion control may include the following steps S3.1 to S3.5:

[0156] S3.1. Path planning output. The RRT algorithm is used to generate the optimal path for the robot from the starting position to the target position, which is usually represented by a linear curve trajectory in a coordinate system. The path is usually a continuous set of points that represent the points that the robot's end effector (such as the end of the robotic arm) needs to pass through.

[0157] S3.2, optimal path discretization. In order to control the movement of the robot arm, it is first necessary to discretize the planned path, that is, to decompose each point on the path into a series of discrete states or postures, each state usually including the position information (position and posture) of the end effector.

[0158] S3.3, inverse dynamics solution. The angle and acceleration of each joint of the robot arm are calculated through the target position information of the end effector. Assuming that the robot is a multi-joint manipulator, the position and posture of the end effector are controlled by the joint angle, so it is necessary to convert it through the inverse dynamics method.

[0159] The specific steps of inverse dynamics conversion include: given the position and posture of the end effector, solving the multivariable nonlinear equations of inverse dynamics through numerical solution to obtain the joint angles corresponding to the position and posture. For complex robotic arms, there may be multiple solutions or no solution, so the solution process needs to be constrained and optimized to ensure the possibility of a solution.

[0160] S3.4, Joint velocity and acceleration control. In order to ensure that the robot arm can smoothly and accurately track the path, it is usually necessary to further calculate the angles and accelerations of the joints. Specifically, the path is smoothed by a trajectory generation algorithm (such as spline interpolation), and the velocity and acceleration between each discrete point are calculated. During the calculation process, the following two constraints must also be met:

[0161] ① Speed ​​planning: Smooth the speed curve according to the discrete points of the path and the maximum speed limit of the robot to ensure that the joint movement does not exceed the speed.

[0162] ② Acceleration planning: Ensure that the acceleration does not exceed the physical limitations of the robot joints to avoid excessive impact.

[0163] S3.5, trajectory tracking and control. The joint instructions (position, speed, acceleration) are input into the robot control system, and the joint angles are continuously adjusted through real-time feedback and control algorithms (such as PID control model prediction model) to ensure that the robot arm performs tasks smoothly along the predetermined trajectory.

[0164] In some embodiments, in order to further improve the accuracy of the operation, visual servo control is performed through Dynamic Window Approach (DWA), and the robot's motion parameters are dynamically adjusted based on real-time visual feedback during the task execution, so that the robot can accurately aim at the target object. The visual data collected by the RGB-D camera can capture the position changes of the target object in real time, ensuring high-precision hand-eye coordination in dynamic scenes. This hand-eye coordination process is further optimized through the Eye-in-Hand Calibration technology, thereby improving the robot's flexibility and response speed in complex tasks.

[0165] S4. Adaptive task execution and force feedback. During the operation, the robot uses force sensors to monitor the contact force with the workpiece in real time and adjusts the operation deviation (such as the operation force) to avoid damage to the workpiece. Through model predictive control (MPC) for closed-loop control, the path and action are dynamically adjusted according to the feedback from the sensor to ensure the accuracy of the operation task.

[0166] In some embodiments, MPC predicts future system behavior based on multi-sensor data input and adjusts control input in real time to ensure the accuracy of the operation path. In addition, this closed-loop control automatically compensates for errors during operation to ensure execution with micron-level precision.

[0167] S5. Reinforcement learning and strategy optimization. When executing tasks, reinforcement learning is performed through the DDPG algorithm to continuously optimize the operation strategy. The initial task generates a preliminary strategy through video imitation learning, and adaptively adjusts and optimizes it according to environmental changes in actual operation. For example, when the workpiece offset is detected, the operation path is automatically adjusted to ensure the successful execution of the task. In addition, through the online adaptive adjustment mechanism, the robot can adjust its behavior according to real-time data during operation, reduce human intervention, and improve operation efficiency and task completion rate. The DDPG algorithm based on the Actor-Critic architecture is adopted to enable the robot to make the optimal strategy in the high-dimensional action space by simultaneously training the policy network (Actor) and the value network (Critic).

[0168] Specifically, the reinforcement learning process of the embodiment of the present application includes the following steps:

[0169] S5.1. In the initial stage of training, the strategy model is initialized by imitating the operations of human experts through the behavior cloning (BC) technology.

[0170] This process uses supervised learning methods to allow the robot to learn the expert's control strategy through historical operation videos and demonstration data. Further, step 5.1 mainly includes the following steps S5.1.1~S5.1.6:

[0171] S5.1.1. Data collection and preparation.

[0172] First, collect demonstration data of the robot's operation. This data usually comes from recordings of human experts performing tasks, including:

[0173] Operation video: records visual information (such as camera images or depth images) of the robot while performing a task, which is used to provide visual input of the robot's environment.

[0174] Control instructions: robot control signals corresponding to the operation video, usually including joint angles, end effector position, speed, etc. Control instructions are real-time input when the robot performs an operation, indicating the state and action of the robot at each moment.

[0175] These demonstration data will be used as training data for supervised learning to train a policy network (usually a deep neural network).

[0176] S5.1.2. Data preprocessing and feature extraction.

[0177] In order for the network to learn effectively, the data needs to be preprocessed and feature extracted. This process includes:

[0178] ① Image data processing: extract key frames from the video and convert them into a format suitable for neural network input (such as normalized pixel values, size scaling, etc.).

[0179] ②Time series processing: align control instructions with corresponding time steps to ensure that each visual input is paired with the correct action label.

[0180] ③Data enhancement: Data enhancement techniques such as rotation, translation, and mirroring can be used to increase data diversity and improve training results.

[0181] S5.1.3. Train the supervised learning model.

[0182] After the data is prepared, a supervised learning method is used to train a neural network, usually a convolutional neural network (CNN) combined with a fully connected layer (FC), to imitate the control strategy of human experts. The goal of training is to learn a mapping relationship by minimizing the error between the predicted action and the expert action:

[0183] Input: Visual information of the current environment (such as images or video frames).

[0184] Output: control instructions (such as robot joint angles, position of the end effector, etc.).

[0185] During the training process, each pair of input and label in the expert demonstration data is used to optimize the network's weight parameters so that the neural network can generate control instructions similar to those of the expert. The commonly used loss function is the mean square error (MSE), which is used to measure the gap between the predicted action and the expert action.

[0186] S5.1.4. Strategy model initialization.

[0187] After completing the supervised learning training, a preliminary strategy model is obtained, that is, a neural network that can imitate the behavior of experts. This model can be used as an initialization strategy for the reinforcement learning process, providing a more accurate initial strategy in the subsequent reinforcement learning stage to avoid starting from scratch.

[0188] S5.1.5. Strategy optimization based on DDPG.

[0189] After the behavior cloning is completed, the Deep Deterministic Policy Gradient (DDPG) algorithm is used for further policy optimization. The DDPG algorithm is a reinforcement learning algorithm suitable for continuous action space. It optimizes the strategy through interaction with the environment. The reinforcement learning model can be further improved through exploration and feedback based on the initial behavior cloning strategy.

[0190] S5.1.6. Strategy adjustment and iteration.

[0191] Although the initial strategy generated by behavioral cloning technology can speed up training, it may still be imperfect. As the robot continues to interact with the environment in actual operation, the reinforcement learning model will gradually adjust and optimize the strategy through experience accumulation and exploration, and ultimately enable the robot to show higher adaptability and operational efficiency in a variety of different tasks and environments (the model obtained by optimizing the initial strategy model during this period is the second strategy model).

[0192] This method not only speeds up the training of the model, but also significantly reduces the random exploration process of reinforcement learning during the training phase, thereby improving operational efficiency.

[0193] In some embodiments, meta-reinforcement learning can also be used to enable the system to quickly generalize across multiple tasks, thereby improving its adaptability in unknown tasks.

[0194] In order to further improve the robot's ability to perform in diverse tasks, multi-task learning and policy sharing mechanisms are adopted. By conducting joint training in multiple task scenarios, the system can learn the common features between tasks, so that it has stronger generalization capabilities when facing new tasks. In specific implementation, the system uses a parameter-sharing policy network so that knowledge between different tasks can be transferred to each other. For example, the precise grasping strategy mastered by the robot when learning assembly tasks can also be applied to similar precision welding operations. This sharing mechanism not only improves training efficiency, but also significantly reduces the demand for training data, thereby improving the task expansion capabilities of the overall system.

[0195] In order to ensure robustness in long-term operation, the embodiment of the present application introduces a periodic evaluation mechanism in reinforcement learning, so that the robot system can automatically update the strategy model to adapt to environmental changes and task requirements.

[0196] Specifically, the robot's operational performance is evaluated by combining reinforcement learning and rule-based exception handling, and the policy model is automatically updated based on this. The following steps S5.1.7 to S5.1.8 are the detailed evaluation and policy update process:

[0197] S5.1.7. Evaluate the robot’s operational performance.

[0198] To ensure the long-term robustness and adaptability of the system, the system regularly evaluates the robot's operational performance in the following ways:

[0199] ①Task completion evaluation: The system evaluates the robot's performance by checking whether it has successfully completed the task. For example, when the robot completes a handling, grasping, or welding task, the system verifies the task results. If the task is successful, it is considered a good performance; if the task fails (such as grasping failure or welding deviation), it is considered a poor performance.

[0200] ② Error and deviation monitoring: The system also monitors errors and deviations during robot operation. For example, in a grasping task, the system records the difference between the robot's grasping position and the target position; in a welding task, the system detects whether the welding point deviates from the predetermined trajectory. Through the size of the error and the degree of deviation, the system can evaluate the execution quality of the robot.

[0201] ③ Feedback loop analysis: The system evaluates the robot's operation through sensory feedback (such as sensor data, visual information, etc.). For example, force sensors are used to monitor force feedback during grasping, or camera feedback is used to check the quality of welding parts. If the feedback is abnormal, the system will mark the operation as failed and start the error recovery process.

[0202] S5.1.8. Automatically update the policy model.

[0203] When the robot encounters failure or poor performance while performing a task, the system will automatically adjust and update the policy model to ensure that the robot can better cope with future tasks. The steps for policy update are as follows S5.1.8.1:

[0204] S5.1.8.1, Failure detection and recording: When the robot encounters a mission failure or abnormal operation, the system will automatically detect the cause of the failure through sensor data, mission completion status and other information. The cause of the failure may be due to environmental changes, improper robot control, sensor failure, etc. The system will automatically record the detailed information of these failure events, including the environmental status at the time of failure, operating parameters, mission type, etc.

[0205] S5.1.8.2, Rule-based exception handling: Once a task failure is detected, the system will first perform preliminary exception handling using preset rules. For example, if the robot finds that the mass of an object has changed while grasping (such as the object sliding, weight change, etc.), the system may try to adjust the grasping force or position to solve the problem. These rules are designed based on experience and are used to quickly respond to some common operation failure scenarios.

[0206] S5.1.8.3, Reinforcement learning strategy optimization: After rule processing, the system will combine reinforcement learning to update the strategy. Specifically, the system will use previous failure records as new training data and update the strategy model based on this data. In this way, the system can continuously optimize the robot's behavior pattern in the event of failure. For example, when the robot fails in a task, the system will introduce "failure" as a feedback signal in reinforcement learning, and adjust the strategy through algorithms such as policy gradients, so that the robot can avoid repeating mistakes in similar tasks in the future.

[0207] For example, suppose the robot damages an object due to excessive grasping force during a grasping task. After the system detects this problem, it will record the failure and feed it back to the reinforcement learning module. The next time the grasping task is performed, the reinforcement learning model will adjust the grasping force strategy to avoid similar problems.

[0208] S5.1.8.4, Continuous Update and Adaptive Optimization: Each updated policy model will be re-evaluated and tested. If the robot successfully completes the task, the system will further verify whether the new policy is effective and continue to update the model. As the robot's operation continues to accumulate experience, the policy model will gradually optimize and become more accurate and robust.

[0209] For example, suppose the robot fails to accurately grasp and successfully move an object due to visual errors during a grasping task. The system first detects the failure to grasp the object through the visual sensor and records the reason for the failure. Then, the system makes preliminary adjustments based on the preset exception handling rules, such as automatically adjusting the grasping position or force.

[0210] The system then further optimizes the policy model through reinforcement learning. Assuming that the cause of failure is the deviation of the grasping position due to visual positioning error, the system inputs the failed operation as negative feedback to the reinforcement learning module and updates the policy network to reduce the impact of visual error. For example, the model may add a strategy of collecting multiple views and comprehensively judging the position of the object, thereby improving the grasping success rate in future tasks.

[0211] S6. Data upload and cloud collaboration. Task execution data is uploaded to the cloud platform, and cloud computing platform and distributed collaboration mechanism are introduced to conduct big data analysis and strategy optimization, and the optimized model is synchronized back to the local robot. The operator can remotely view the task progress through the cloud monitoring interface and adjust the task parameters in real time if necessary.

[0212] In the daily operation of the system, various sensors (such as RGB-D cameras, lidars, force sensors) and robot actuators generate a large amount of operational data. In order to better manage this data, the system adopts a cloud data storage architecture. First, the sensor data is preprocessed by edge computing devices (such as NVIDIA Jetson AGX Orin) to ensure that the data uploaded to the cloud has been preliminarily screened and denoised, thereby reducing the data transmission bandwidth. The preprocessed data is uploaded to the cloud database (such as AWS S3 or Azure Blob Storage) in real time through the MQTT protocol or ROS2 communication protocol, and is classified and stored according to the task type and timestamp. Through the cloud storage system, operators can retrieve historical data for analysis at any time, which is convenient for equipment maintenance and fault diagnosis.

[0213] In order to further improve the robot's execution efficiency in complex tasks, this module integrates big data analysis and machine learning algorithms to conduct in-depth analysis of data collected in the cloud. Through distributed data processing frameworks such as Hadoop and Spark, the system can process massive amounts of operation data in parallel, extract operation patterns, identify potential bottlenecks, and provide data support for the reinforcement learning module. For example, in a multi-robot system, the system can analyze the operation data of each robot, identify success and failure patterns, and generate improved strategies. Based on data analysis, the system will regularly update the policy model in the cloud and distribute the optimized model to each robot terminal, so that it can show higher efficiency and accuracy in subsequent tasks.

[0214] Specifically, step S6 includes the following steps S6.1 to S6.7:

[0215] S6.1. Data collection and upload. When performing tasks, the robot collects data in real time through a variety of sensors (such as RGB-D cameras, lidar, force sensors, etc.), including environmental perception data, task execution data, and status feedback data. These data are processed by edge computing devices and uploaded to the cloud platform.

[0216] S6.2, Data storage and management. The cloud platform stores the collected data in categories, including task execution logs, sensor data, and exception records. A distributed storage system (such as AWSS3) is used to ensure data security and scalability for subsequent analysis.

[0217] S.3. Data analysis and feature extraction. Through distributed data processing tools (such as Apache Spark), batch analysis of stored data is performed to extract key features of task execution, such as path planning efficiency, environmental changes, and abnormal patterns. These features provide a basis for subsequent model optimization.

[0218] S6.4. Continuous optimization of the strategy model. Using machine learning algorithms such as reinforcement learning, the robot's task execution strategy is optimized by continuously updating the strategy model. During the optimization process, the robot's decision-making and execution strategy in actual tasks are continuously adjusted through the features extracted through data analysis.

[0219] S6.5, Feedback and Verification. The optimized model will be verified through actual tasks, and feedback information such as success rate and execution time will be collected, and the results will be fed back to the cloud. Based on these verification data, the model will be further iterated and optimized.

[0220] S6.6 Main dimensions of data sources. Big data analysis is mainly based on robot task logs, sensor data, operation strategies, and environmental change data, which together form the basis for model optimization. Robots continuously learn from different tasks and feed back their experience to the cloud platform.

[0221] S6.7 Continuous optimization example in application scenarios. In precision assembly or welding tasks, by analyzing force and visual data, the robot's operation path and force adjustment are optimized to improve the accuracy and efficiency of task completion. Reinforcement learning and optimization algorithms can help robots gradually improve their performance in different tasks.

[0222] In some embodiments, in order to achieve efficient management of intelligent production lines, the cloud module can also support remote monitoring and task scheduling. The operator can view the current status and operation of the robot through a real-time monitoring interface based on WebSocket. When a robot operation deviation or equipment abnormality is detected, the system will alert the operator through an abnormal alarm mechanism (such as SMS or email notification). Through the cloud control panel, the operator can remotely adjust task priorities, reallocate tasks, and intervene in robot operations in real time. Combined with a rule-based task scheduling algorithm, the system can dynamically adjust the task allocation of each robot according to production needs, ensuring efficient collaboration in a multi-task, multi-factory environment.

[0223] Specifically, the cloud platform usually communicates with the robot through an API interface to obtain its status information. The robot's status information is uploaded to the cloud in real time through wireless networks (such as Wi-Fi, 5G, LPWAN, etc.). The platform obtains the current working status of the robot by regularly or in real time querying the robot status data. The cloud platform can dynamically adjust tasks in the following ways:

[0224] ① Real-time monitoring: Make timely task adjustment decisions by monitoring robot status, task progress and other data. Robot status includes: Standby status: The robot is in standby mode and ready to perform tasks. Executing status: The robot is performing tasks or operations. Paused status: The robot is paused while performing tasks and may be waiting for instructions or handling exceptions. Fault status: The robot has a fault and cannot work properly. Charging status: The robot is charging and automatically enters this state when the battery is low. Completion status: The robot successfully completes the task and is ready to accept the next task.

[0225] ②Task scheduling and allocation: Adjust the priority of tasks or reallocate tasks according to the status and needs of the robot.

[0226] ③ Remote control: On the cloud platform, administrators can manually adjust task parameters or directly send new task instructions to the robot.

[0227] ④ Intelligent optimization: The platform can automatically optimize task scheduling through algorithms based on factors such as robot status and environmental changes to ensure that tasks are completed on time and efficiently.

[0228] In order to improve the intelligence level of the multi-robot system, the embodiment of the present application introduces a distributed learning and model sharing mechanism. Through federated learning, robots in each factory can learn and optimize task strategies locally without uploading data to the cloud, thereby protecting data privacy. Under the federated learning framework, the robot will regularly upload locally trained model updates to the cloud. After the system aggregates the model in the cloud, it will distribute the optimized global model back to each terminal device. This process ensures that each robot can continuously improve its task execution capabilities in different operating environments, while realizing knowledge sharing between robots, avoiding repeated learning, and improving learning efficiency.

[0229] Furthermore, in some embodiments, the data processing process for implementing knowledge sharing between different robots through federated learning technology mainly includes the following steps:

[0230] (1) Data collection and local processing. When each robot performs a task, it collects local environmental data, sensor data, and other information. This data includes sensor information such as vision, sound, temperature, and distance, as well as status data during the task execution process (such as position, speed, load, etc.).

[0231] The robot will perform preliminary data processing and analysis locally, converting useful information into features for model training.

[0232] (2) Local model training. Each robot trains a local model based on local data using deep learning, machine learning, and other algorithms. For example, a robot can identify obstacles during task execution or optimize path planning by training a deep neural network (DNN).

[0233] The training of local models is performed independently, and the data does not leave the robot's device, protecting data privacy and security.

[0234] (3) Model update and encrypted transmission.

[0235] After the local model training is completed, the robot will only upload the encrypted and compressed model parameters (such as weights, gradients, etc.) instead of the original data. This can ensure data privacy and avoid leaking sensitive information.

[0236] These encrypted model parameters will be sent to the cloud platform or the coordination node between multiple robots for global model aggregation.

[0237] (4) Global model aggregation. The cloud platform or coordination node receives model parameters from different robots and aggregates them. Common aggregation methods include weighted averaging or other optimization algorithms, which assign different weights to each robot based on its model quality or data volume. The aggregated global model represents the experience and knowledge of multiple robots in different environments, tasks, and states.

[0238] (5) Global model transmission and local update. The aggregated global model will be sent back to each robot, and the robot will use this global model to update its local model. In this way, each robot can use the knowledge of other robots to improve its task execution ability.

[0239] After receiving updates from the global model, the robot will adjust its decision-making strategy, path planning, etc. based on the new global knowledge.

[0240] (6) Iteration and continuous optimization. This process is a continuous iterative process. As the robot continues to perform tasks, acquires new data and conducts local training, the accuracy and generalization ability of the model are gradually improved.

[0241] Each robot is constantly optimizing and adjusting during task execution, and the cloud platform will continue to receive model parameters uploaded by each robot for new aggregation to form a more accurate global model, promoting the sharing and improvement of robot group knowledge.

[0242] (7) Collaboration and knowledge sharing. Through federated learning, robots can share knowledge, avoiding the privacy risks brought about by centralized data storage and processing, while improving the overall intelligence level of the system.

[0243] Federated learning can also enable collaborative work between robots. For example, if a robot encounters an obstacle or problem in a certain task, after knowledge sharing, other robots can obtain similar experience in advance, avoid repeated trial and error, and improve work efficiency.

[0244] Since the system needs to transmit and store a large amount of production data in the cloud, data security is of vital importance. To ensure the security of data transmission, this module uses TLS encryption and VPN channels to prevent data from being stolen or tampered with during transmission. In addition, the system uses role-based access control (RBAC) and multi-factor authentication (MFA) to ensure that only authorized operators and administrators can access cloud data and control panels. Data is encrypted using AES-256 during storage to ensure that sensitive data is not threatened when the storage device is stolen or leaked. The system also regularly performs security vulnerability scans and penetration tests to ensure the security of the cloud platform.

[0245] S7, system self-check and task completion. After the task is completed, the system automatically performs a status self-check to ensure that each module is operating normally and saves the operation log. The robot is on standby, waiting for new task instructions or adjusting the task strategy to adapt to subsequent operation requirements.

[0246] In some embodiments, first, to ensure the seamless coordination of various modules, the system adopts a modular integrated architecture. ROS2 (Robot Operating System 2) is used as the system integration framework to uniformly manage the data communication, task scheduling and coordinated operation of various modules. The distributed architecture of ROS2 can effectively manage multi-sensor data streams (such as RGB-D cameras, LiDAR, force sensors) and control signals to ensure seamless connection between different hardware devices and software modules. During the system integration process, each module is deployed through a Docker container to achieve environmental isolation, reduce software dependency conflicts, and thus improve the stability of the overall system. In the functional verification stage, the system adopts a behavior-driven development (BDD) method to verify whether the functions of each module meet the design requirements by writing a series of functional test cases. For example, when verifying the motion control module, the robot arm's motion smoothness and accuracy under different load conditions will be tested; when verifying the multimodal perception module, its object recognition accuracy under complex lighting conditions will be tested. These test results will provide data support for further optimization of the system.

[0247] To ensure the stability and reliability of the system in a complex industrial environment, the system integration and test verification module has designed a series of performance testing and stress testing schemes. In the performance test, the focus is on evaluating the system's response time, sensor data processing efficiency, path planning speed, and execution accuracy under high load. For example, by simulating high-frequency data input (such as simultaneous data collection from multiple lidars and cameras), the system's processing capabilities in a busy production line are tested. In the stress test, the system will test the system's extreme tolerance by increasing task complexity, increasing the sampling frequency of sensor data, and introducing random environmental interference. Through these tests, the system can identify potential performance bottlenecks and perform necessary optimizations before integration.

[0248] To further improve the efficiency and accuracy of system integration testing, digital twin technology can be introduced. By creating a virtual model that corresponds to the real production environment, the system can simulate real operation tasks in a virtual environment and conduct comprehensive testing and tuning of each module. During the integration test, the digital twin system can reflect the behavior of the robot in actual operation in real time and perform simulation evaluation on it. For example, by testing assembly tasks in a digital twin environment, the system can pre-detect problems in path planning and operation processes, so that they can be fixed before deployment. This method significantly reduces the debugging time and cost during the actual deployment process.

[0249] In order to ensure that the system remains stable after updates and maintenance, the system integration and test verification module adopts a continuous integration (CI) and regression testing strategy. After each system update (such as algorithm adjustment, strategy model optimization), a series of regression tests will be automatically triggered to verify whether the new version has introduced any functional defects or performance degradation. Through tools such as Jenkins or GitLab CI, the system automatically performs unit testing, integration testing, and end-to-end testing to ensure that code changes do not affect the overall performance of the system. In addition, the system is regularly tested comprehensively through automated test scripts to detect potential problems in a timely manner and ensure the stability of the system in long-term operation.

[0250] In order to evaluate the adaptability of the system in actual industrial applications, the embodiments of the present application design a variety of scenario task verification tests. The test scenarios cover different tasks such as automated assembly, precision welding, quality inspection, etc., to ensure the efficient operation of the system in multiple tasks and multiple scenarios. By adjusting environmental parameters (such as temperature, humidity, lighting conditions, etc.) and task complexity, the system can verify its stability under extreme conditions. In addition, the robot's collaborative work ability is tested in a multi-task parallel scenario to evaluate the system's performance in task scheduling and resource management. The test results will be fed back to the cloud for data management, which will be used to further optimize the reinforcement learning strategy and task execution efficiency.

[0251] In summary, the embodiments of the present application have at least the following beneficial effects:

[0252] 1. This invention greatly improves the robot's perception of complex environments by integrating the multimodal perception technology of RGB-D cameras, laser radars, and force sensors. The deep fusion of multi-source data using PointFusion and PointNet++ technologies ensures that the robot can still accurately identify and locate workpieces under lighting changes and occlusions, thereby achieving more stable operation. In addition, the integration of real-time 3D environment modeling and SLAM technology enables the robot to have the ability to adaptively adjust its path in a dynamic environment.

[0253] 2. The present invention adopts the RRT algorithm for efficient path planning and combines the inverse dynamics model to realize the precise motion control of the robot arm. In addition, the system integrates visual servo control (DWA) and model predictive control (MPC) technology, and through the closed-loop feedback of multi-sensor data, the robot can adjust the path and action in real time during operation. This adaptive control mechanism not only improves the operation accuracy, but also reduces the task execution time, greatly improving the production efficiency.

[0254] 3. By introducing deep reinforcement learning (DDPG) and behavior cloning technology, the present invention enables the robot to have the ability of rapid self-learning and adaptive adjustment. The system generates the initial strategy through video imitation learning and continuously optimizes it through reinforcement learning in actual operation. During the task execution process, the robot can adjust the strategy based on real-time sensor data to achieve rapid adaptation to different task scenarios. This not only reduces the dependence on manual intervention, but also improves the execution efficiency of the robot in diversified tasks.

[0255] See also Figure 3 The embodiment of the present application also provides a multi-modal perception humanoid robot motion adaptive control system, which can implement the above multi-modal perception humanoid robot motion adaptive control method, and the system includes:

[0256] The multimodal sensing module 101 is used to simultaneously collect data in the environment through an RGB-D camera, a laser radar and a force sensor and perform preprocessing to obtain first multimodal data; wherein the first multimodal data includes first image data, first point cloud data and first force data;

[0257] A data fusion and environment modeling module 102, configured to perform data fusion on the first image data and the first point cloud data using a deep learning algorithm, and to construct a three-dimensional model of the operation scene in real time according to the result of the data fusion, and to obtain real-time environment data of the three-dimensional model;

[0258] The path planning and control module 103 is used to generate an optimal path according to the real-time environment data, convert the optimal path into joint instructions of the humanoid robot through an inverse dynamics model, and generate a motion control strategy; and is also used to adjust the arm movement of the humanoid robot through visual servo control according to the motion control strategy, so that the humanoid robot performs the task and obtains first operation data;

[0259] An adaptive task execution and force feedback module 104 is used to adjust the path and action of the humanoid robot in a closed-loop control manner according to the first force data during the operation of the humanoid robot to obtain second operation data;

[0260] The reinforcement learning and strategy optimization module 105 is used to perform reinforcement learning and regular evaluation of the motion control strategy in combination with the first operation data and the second operation data through the DDPG algorithm during the operation of the humanoid robot, so as to continuously optimize the motion control strategy.

[0261] In some embodiments, the system may also include a cloud data management and remote collaboration module 106, which is used to upload the task execution data obtained by the humanoid robot performing the operation task to the cloud platform for storage; wherein the task execution data includes the robot task log, the real-time environment data, the first multimodal data, and the action execution strategy; it is also used to execute the steps of continuously optimizing the action control strategy by using the DDPG algorithm to perform reinforcement learning and regular evaluation on the action control strategy in combination with the first operation data and the second operation data during the operation of the humanoid robot according to the task execution data stored on the cloud platform; it is also used to perform big data analysis based on the task execution data stored on the cloud platform, and optimize the action control strategy according to the results of the big data analysis; it is also used to synchronize the optimized action control strategy model to the humanoid robot.

[0262] In some embodiments, the system may further include a system integration and test verification module 107 for integrating the functions of each module and uniformly managing data communication, task scheduling and coordinated operations of each module.

[0263] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0264] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above multi-modal perception humanoid robot motion adaptive control method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0265] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0266] See also Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0267] The processor 201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0268] The memory 202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 202 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 202, and the processor 201 calls and executes the multi-modal perception humanoid robot motion adaptive control method of the embodiment of this application;

[0269] Input / output interface 203, used to implement information input and output;

[0270] The communication interface 204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0271] Bus 205 , which transmits information between various components of the device (e.g., processor 201 , memory 202 , input / output interface 203 , and communication interface 204 );

[0272] The processor 201 , the memory 202 , the input / output interface 203 and the communication interface 204 are connected to each other in communication within the device via the bus 205 .

[0273] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned multi-modal perception humanoid robot motion adaptive control method.

[0274] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0275] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0276] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0277] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0278] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0279] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0280] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0281] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0282] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0283] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0284] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0285] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0286] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A multi-modal perception humanoid robot motion adaptive control method, characterized in that: The following steps are involved: The data in the environment is collected simultaneously by an RGB-D camera, a laser radar and a force sensor and preprocessed to obtain first multimodal data; wherein the first multimodal data includes first image data, first point cloud data and first force data; Using a deep learning algorithm to perform data fusion on the first image data and the first point cloud data, and constructing a three-dimensional model of the operation scene in real time according to the result of the data fusion, and obtaining real-time environmental data of the three-dimensional model; Generate an optimal path according to the real-time environmental data, convert the optimal path into joint instructions of the humanoid robot through an inverse dynamics model, and generate a motion control strategy; According to the motion control strategy, adjusting the arm movement of the humanoid robot through visual servo control so that the humanoid robot performs the task and obtains first operation data; During the operation of the humanoid robot, according to the first force sense data, a path and a movement of the humanoid robot are adjusted by a closed-loop control method to obtain second operation data; During the operation of the humanoid robot, the motion control strategy is reinforced learned and regularly evaluated by using the DDPG algorithm in combination with the first operation data and the second operation data, so as to continuously optimize the motion control strategy.

2. The method according to claim 1, characterized in that The method of simultaneously collecting data in the environment by using an RGB-D camera, a laser radar, and a force sensor and performing preprocessing to obtain first multimodal data includes the following steps: The RGB-D camera collects the color image and depth information in the environment to obtain an RGB image; Acquire three-dimensional point cloud data in the environment by using a laser radar; During the working process of the humanoid robot, the contact force of the humanoid robot on the target workpiece is collected through the force sensor to obtain force sensor data; Performing gamma correction and white balance adjustment on the RGB image to obtain depth image data; Performing bilateral filtering on the first processed image to obtain first image data; Downsampling the three-dimensional point cloud data to obtain key point cloud data; Using a random sampling consistency algorithm to remove outliers and noise data in the key point cloud data to obtain first point cloud data; Kalman filtering is performed on the force sensor data to obtain first force data.

3. The method according to claim 1, characterized in that The step of fusing the first image data and the first point cloud data using a deep learning algorithm, constructing a three-dimensional model of the operation scene in real time according to the result of the data fusion, and acquiring real-time environmental data of the three-dimensional model includes the following steps: Using a three-dimensional convolutional neural network to extract three-dimensional features of the first image data and the first point cloud data; According to the three-dimensional features, performing data fusion processing on the first image data and the first point cloud data to obtain first fused data; According to the first fusion data, real-time positioning of the humanoid robot is performed in a dynamic environment, and a three-dimensional model is constructed; wherein the three-dimensional model includes a three-dimensional map and an object model; Converting the first point cloud data into a voxel map; continuously updating the three-dimensional model according to the voxel map; According to the three-dimensional model, real-time environmental data of the humanoid robot during its working process is obtained.

4. The method according to claim 1, characterized in that: The method of generating an optimal path according to the real-time environmental data, converting the optimal path into joint instructions of the humanoid robot through an inverse dynamics model, and generating a motion control strategy comprises the following steps: Using an RRT algorithm, based on the three-dimensional environment map and the positioning of the end effector of the humanoid robot, an optimal path of the humanoid robot from a starting position to a target position is generated; Discretize the optimal path to obtain position information of each path point of the end effector on the optimal path; According to the position information, solving a first joint angle and a first joint velocity of the end effector at the path point by inverse dynamics; According to the joint angle and the joint speed, a path is smoothed by a trajectory generation algorithm, and a second speed and a second acceleration between each of the path points are calculated; generating a joint instruction according to the position information, the first joint angle, the first joint velocity, the second velocity and the second acceleration; A motion control strategy is generated according to the joint instructions.

5. The method according to claim 1, characterized in that During the operation of the humanoid robot, adjusting the path and action of the humanoid robot by a closed-loop control method according to the first force sense data to obtain second operation data includes the following steps: According to the first force sense data, the humanoid robot detects the contact force between the humanoid robot and the target workpiece to obtain a contact force monitoring result; Predicting the future behavior of the humanoid robot according to the contact force monitoring result by a model predictive control algorithm to obtain a behavior prediction result; Adaptively adjusting the path and action of the humanoid robot according to the behavior prediction result; The humanoid robot is controlled to perform an operation according to the adjusted path and action to obtain second operation data.

6. The method according to claim 1, characterized in that During the operation of the humanoid robot, the action control strategy is reinforced learned and periodically evaluated by combining the first operation data and the second operation data through the DDPG algorithm to continuously optimize the action control strategy, including the following steps: Generate a preliminary policy model for parameter sharing by imitating historical operation videos and demonstration data through behavioral cloning technology; Based on a deep deterministic policy gradient algorithm, jointly train the preliminary policy model in multiple scenarios, and optimize the policy of the preliminary policy model according to the first operation data and the second operation data to obtain a second policy model; Performing an operation performance evaluation according to the task completion degree of the humanoid robot during the working process to obtain an operation performance evaluation result; The second strategy model is adaptively adjusted according to the operation performance evaluation result to achieve continuous optimization of the action control model.

7. The method according to claim 1, characterized in that The method further comprises the following steps: Uploading the task execution data obtained by the humanoid robot performing the operation task to the cloud platform for storage; wherein the task execution data includes the robot task log, the real-time environment data, the first multimodal data, and the action execution strategy; According to the task execution data stored in the cloud platform, performing, during the operation of the humanoid robot, a step of performing reinforcement learning and regular evaluation on the motion control strategy by combining the first operation data and the second operation data through a DDPG algorithm to continuously optimize the motion control strategy; Performing big data analysis on the task execution data stored on the cloud platform, and optimizing the motion control strategy according to the result of the big data analysis; The optimized motion control strategy model is synchronized to the humanoid robot.

8. A multi-modal sensing humanoid robot motion adaptive control system, characterized in that: The system comprises: A multimodal perception module, used to simultaneously collect data in the environment through an RGB-D camera, a laser radar, and a force sensor and perform preprocessing to obtain first multimodal data; wherein the first multimodal data includes first image data, first point cloud data, and first force data; A data fusion and environment modeling module, configured to perform data fusion on the first image data and the first point cloud data using a deep learning algorithm, and to construct a three-dimensional model of the operation scene in real time according to the result of the data fusion, and to obtain real-time environment data of the three-dimensional model; A path planning and control module, used to generate an optimal path according to the real-time environmental data, convert the optimal path into joint instructions of the humanoid robot through an inverse dynamics model, and generate a motion control strategy; and also used to adjust the arm movement of the humanoid robot through visual servo control according to the motion control strategy, so that the humanoid robot performs the task and obtains first operation data; an adaptive task execution and force feedback module, configured to adjust the path and action of the humanoid robot in a closed-loop control manner according to the first force data during the operation of the humanoid robot, so as to obtain second operation data; The reinforcement learning and strategy optimization module is used to perform reinforcement learning and regular evaluation of the motion control strategy in combination with the first operation data and the second operation data through the DDPG algorithm during the operation of the humanoid robot, so as to continuously optimize the motion control strategy.

9. The system according to claim 8, characterized in that The system further comprises: A cloud data management and remote collaboration module is used to upload the task execution data obtained by the humanoid robot when performing the operation task to the cloud platform for storage; wherein the task execution data includes the robot task log, the real-time environment data, the first multimodal data, and the action execution strategy; it is also used to execute the steps of continuously optimizing the action control strategy by using the DDPG algorithm to perform reinforcement learning and regular evaluation on the action control strategy in combination with the first operation data and the second operation data during the operation of the humanoid robot according to the task execution data stored on the cloud platform; it is also used to perform big data analysis based on the task execution data stored on the cloud platform, and optimize the action control strategy according to the results of the big data analysis; it is also used to synchronize the optimized action control strategy model to the humanoid robot.

10. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

Images