A model reasoning-based humanoid robot control method and related device

Abstract

The application discloses a humanoid robot control method based on model reasoning and related equipment, and the method comprises the following steps: collecting multi-modal data dynamically through a target robot end, transforming the multi-modal data to generate multi-modal transformed data; dynamically monitoring the network communication state of the target robot end, and according to the network communication state and the data sensitivity of the multi-modal transformed data, inferring the multi-modal transformed data through a lightweight inference service model to output an inference result; the lightweight inference service model is obtained through full-layer parameter transformation and federated learning collaborative training; the inference result is inversely transformed through the target robot end to obtain an action control instruction, and the target humanoid robot is controlled according to the action control instruction. The embodiment of the application can realize robot automation control based on multi-modal data input, improve the robot control efficiency and accuracy. The application can be widely applied to the field of robot control technology.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a model-based reasoning-based humanoid robot control method and related equipment. Background Technology

[0002] Currently, target robots are increasingly widely used in fields such as industrial automation, home services, and medical care. However, their technological implementation faces the dual challenges of real-time performance and data security. Existing solutions in related technologies have significant limitations:

[0003] 1) The motion control of the target robot requires extremely high timeliness, with a typical motion control cycle needing to be kept within 10-20ms. However, traditional cloud-based inference-based AI model deployment methods have the following problems:

[0004] Uncontrollable network latency: Even though the theoretical latency of 5G networks is as low as 1ms, in actual industrial environments, signal interference and bandwidth fluctuations may cause latency to exceed 50ms, leading to robot lag or even loss of control.

[0005] Network outages pose a fatal risk: In critical scenarios such as industrial production lines, network interruptions can cause robot systems to stagnate, leading to production accidents or economic losses.

[0006] 2) In traditional solutions, lightweight models (such as MobileNet) are difficult to handle the complex multimodal fusion tasks of robots, and the risk of model stealing attacks still exists.

[0007] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0008] The main objective of this application is to propose a humanoid robot control method and related equipment based on model reasoning.

[0009] To achieve the above objectives, one aspect of this application proposes a humanoid robot control method based on model reasoning, the method comprising the following steps:

[0010] Multimodal data is dynamically collected from the target robot, and the multimodal data is transformed to generate multimodal transformed data.

[0011] The network communication status of the target robot is dynamically monitored. Based on the network communication status and the data sensitivity of the multimodal transformation data, a lightweight inference service model is used to infer the multimodal transformation data and output the inference result. The lightweight inference service model is obtained through full-layer parameter transformation and federated learning collaborative training.

[0012] The inference result is inversely transformed by the target robot to obtain motion control commands, and the target humanoid robot is controlled according to the motion control commands.

[0013] In some embodiments, the method further includes:

[0014] After performing full-layer parameter transformation on the public lightweight inference service model through public cloud nodes, the public lightweight inference service model and the permutation matrix are distributed to each robot terminal.

[0015] Each robot receives the public task processing model and stores it locally as a corresponding local inference service model. The local dataset is used to train the local inference service model and generate corresponding model update parameters.

[0016] Each robot terminal transmits the corresponding model update parameters, performs permutation and alignment of the model update parameters according to the permutation matrix, and encrypts and uploads the permuted and aligned model update parameters to the public cloud node. The public cloud node performs global model aggregation based on the encrypted model update parameters uploaded by each private cloud node, determines the aggregation result, and updates the public lightweight inference service model based on the aggregation result.

[0017] The updated public lightweight inference service model is encrypted through the public cloud node and then distributed to each of the robot terminals. Each robot terminal updates its corresponding local inference service model based on the decrypted public lightweight inference service model.

[0018] In some embodiments, the step of dynamically acquiring multimodal data through a target robot end, transforming the multimodal data, and generating multimodal transformed data includes:

[0019] The multimodal data is dynamically acquired through the target robot end using multimodal sensors. The multimodal data includes at least visual image data, tactile signal data, and joint sequence data.

[0020] The target robot performs matrix transformation on the multimodal data according to the permutation matrix to generate the multimodal transformed data.

[0021] In some embodiments, the step of performing full-layer parameter transformation on the public lightweight inference service model via a public cloud node, and then distributing the public lightweight inference service model and the permutation matrix to each robot terminal includes:

[0022] Obtain the model parameter set corresponding to the public lightweight inference service model. The model parameter set includes at least the multi-head attention layer parameters, the forward layer parameters, and the layer normalization layer parameters.

[0023] The permutation matrix is ​​randomly generated, and the dimension of the permutation matrix is ​​consistent with the hidden layer dimension of the public lightweight inference service model.

[0024] Using the permutation matrix, the model parameter set is subjected to linked parameter transformation to obtain a public lightweight inference service model after full-layer parameter transformation;

[0025] The public lightweight inference service model after full-layer parameter transformation and the permutation matrix are distributed to each robot terminal through the public cloud node.

[0026] In some embodiments, the dynamic monitoring of the network communication status of the target robot, and the inference of the multimodal transformation data based on the network communication status and the data sensitivity of the multimodal transformation data using a lightweight inference service model, and the output of the inference result, including:

[0027] When the network communication status is connected, the multimodal transformation data is collaboratively reasoned and analyzed based on the data sensitivity of the multimodal transformation data through the public lightweight reasoning service model and the local reasoning service model of the target robot, and the reasoning result is generated.

[0028] When the network communication status is offline, the multimodal transformation data is inferred through the local inference service model of the target robot to generate the inference result.

[0029] In some embodiments, when the network communication state is connected, collaborative reasoning analysis is performed on the multimodal transformation data based on the data sensitivity of the multimodal transformation data using the public lightweight inference service model and the local inference service model of the target robot, to generate the inference result, including:

[0030] Data sensitivity detection is performed on the multimodal transformation data, and the multimodal transformation data is split into sensitive data and non-sensitive data;

[0031] The sensitive data is inferred using the local inference service model on the target robot to generate a first inference result;

[0032] The non-sensitive data is encrypted and uploaded to the public cloud node through the target robot terminal. The encrypted non-sensitive data is decrypted through the public cloud node. The decrypted non-sensitive data is then inferred using the public lightweight inference service model to generate a second inference result.

[0033] The second inference result is encrypted and sent to the target robot terminal through the public cloud node, and the target robot terminal decrypts the encrypted second inference result to obtain the decrypted second inference result;

[0034] The reasoning result is determined based on the first reasoning result and the decrypted second reasoning result.

[0035] In some embodiments, the step of performing an inverse transformation on the reasoning result through the target robot to obtain motion control commands, and controlling the target humanoid robot according to the motion control commands, includes:

[0036] The inference result is inversely transformed using the permutation matrix stored locally on the target robot to obtain the action control command;

[0037] The target humanoid robot is driven to perform corresponding actions according to the motion control instructions via the target robot end.

[0038] To achieve the above objectives, another aspect of this application proposes a humanoid robot control device based on model reasoning, the device comprising:

[0039] The first module is used to dynamically collect multimodal data through the target robot end, transform the multimodal data, and generate multimodal transformed data;

[0040] The second module is used to dynamically monitor the network communication status of the target robot. Based on the network communication status and the data sensitivity of the multimodal transformation data, it performs inference on the multimodal transformation data through a lightweight inference service model and outputs the inference result. The lightweight inference service model is obtained through full-layer parameter transformation and federated learning collaborative training.

[0041] The third module is used to perform an inverse transformation on the reasoning result through the target robot end to obtain motion control instructions, and to control the target humanoid robot according to the motion control instructions.

[0042] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0043] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0044] The embodiments of this application include at least the following beneficial effects: This application provides a humanoid robot control method, device, electronic device, and storage medium based on model inference. This scheme dynamically collects multimodal data from the target robot, transforms the multimodal data to generate multimodal transformed data, and, based on the network communication status of the target robot and the data sensitivity of the multimodal transformed data, performs inference on the multimodal transformed data through a lightweight inference service model, outputting the inference result. The target robot then performs an inverse transformation on the inference result to obtain motion control commands, and controls the target humanoid robot according to the motion control commands. This application can realize automated robot control based on multimodal data input. Through full-layer parameter transformation and federated learning collaborative training of the lightweight inference service model, it can meet the real-time control cycle requirement of 10-20ms without increasing the computational load on the robot. Based on the network communication status of the target robot and the data sensitivity of the multimodal data, model inference is performed, enabling the robot to maintain real-time task processing capabilities in scenarios with no network or weak network, avoiding job interruptions or safety accidents due to network interruptions, while maintaining data security and improving robot control efficiency and accuracy. Attached Figure Description

[0045] Figure 1 This is a flowchart of a humanoid robot control method based on model reasoning provided in an embodiment of this application;

[0046] Figure 2 This is another flowchart of a model-based reasoning humanoid robot control method provided in the embodiments of this application;

[0047] Figure 3 This is a schematic diagram of the structure of a humanoid robot control device based on model reasoning provided in an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0050] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0051] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0053] The humanoid robot control method based on model reasoning provided in this application relates to the field of information technology. This model reasoning-based humanoid robot control method can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the model reasoning-based humanoid robot control method, but is not limited to the above forms.

[0054] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0055] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0056] Figure 1This is an optional flowchart of a model-based reasoning-based humanoid robot control method provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S103.

[0057] Step S101: Dynamically collect multimodal data through the target robot end, transform the multimodal data, and generate multimodal transformed data;

[0058] Step S102: Dynamically monitor the network communication status of the target robot. Based on the network communication status and the data sensitivity of the multimodal transformation data, infer the multimodal transformation data through a lightweight inference service model and output the inference results. The lightweight inference service model is obtained through full-layer parameter transformation and federated learning collaborative training.

[0059] Step S103: The reasoning result is inversely transformed through the target robot to obtain motion control instructions, and the target humanoid robot is controlled according to the motion control instructions.

[0060] In some embodiments, refer to Figure 2 The above method may also include steps S201 to S204.

[0061] Step S201: After performing full-layer parameter transformation on the public lightweight inference service model through the public cloud node, the public lightweight inference service model and the permutation matrix are distributed to each robot terminal.

[0062] Step S202: Receive the common task processing model through each robot terminal, store it locally as the corresponding local inference service model, train the local inference service model using the local dataset, and generate the corresponding model update parameters.

[0063] Step S203: The corresponding model update parameters are updated by each robot terminal. The model update parameters are permuted and aligned according to the permutation matrix. The permuted and aligned model update parameters are encrypted and uploaded to the public cloud node. The public cloud node performs global model aggregation based on the model update parameters encrypted and uploaded by each private cloud node, determines the aggregation result, and updates the public lightweight inference service model based on the aggregation result.

[0064] Step S204: After encrypting the updated public lightweight inference service model through the public cloud node, the updated model is distributed to each robot terminal. Each robot terminal updates its corresponding local inference service model based on the decrypted public lightweight inference service model.

[0065] In some embodiments, each robot terminal is based on a high-performance computing server to meet the local processing requirements of sensitive data, while public cloud nodes are based on an elastic cloud computing platform to provide computing power acceleration for non-sensitive data.

[0066] Optionally, the public cloud node includes a model development module, which employs a combination of knowledge distillation and quantization compression techniques, as detailed below:

[0067] 1) Knowledge distillation: Transferring knowledge from the original large model to a lightweight architecture, training through a teacher-student network, retaining core reasoning capabilities, while reducing the number of model parameters by more than 80%. One embodiment compresses the visual detection model from 86M parameters to 17M, maintaining high efficiency on embedded devices.

[0068] 2) Mixed-precision quantization: The model weights are quantized from FP32 to FP16 or INT8, further reducing the storage volume by more than 50%. The quantized model can be stored in the robot's embedded memory, adapting to mainstream storage capacities.

[0069] 3) Dynamic weight pruning: By eliminating redundant connections through regularization, the computational cost of the model is reduced by 30%-40% while maintaining accuracy, significantly reducing the time spent on matrix operations.

[0070] The robot utilizes a heterogeneous computing architecture and achieves low-latency inference through three-layer optimization, as detailed below:

[0071] Hardware acceleration layer: Integrates SIMD instruction set or dedicated matrix operation unit to control the calculation latency of permutation matrix multiplication to within 1ms, ensuring real-time performance.

[0072] Algorithm optimization layer: The block matrix decomposition technique is adopted to split large matrix operations into multiple sub-blocks for parallel computation, and hardware threads are used to accelerate the computation and improve efficiency.

[0073] Memory management layer: Design a hierarchical caching mechanism to store frequently accessed permutation matrices and intermediate features in a high-speed cache, reducing memory access latency and improving data reading speed.

[0074] In some embodiments, step S201 may include, but is not limited to, steps S301 to S304.

[0075] Step S301: Obtain the model parameter set corresponding to the public lightweight inference service model. The model parameter set includes at least the multi-head attention layer parameters, the forward layer parameters, and the layer normalization layer parameters.

[0076] Step S302: Randomly generate a permutation matrix, wherein the dimension of the permutation matrix is ​​consistent with the hidden layer dimension of the public lightweight inference service model.

[0077] Step S303: Using the permutation matrix, perform linked parameter transformation on the model parameter set to obtain the public lightweight inference service model after full-layer parameter transformation.

[0078] Step S304: The public lightweight inference service model and permutation matrix after full-layer parameter transformation are distributed to each robot terminal through the public cloud node.

[0079] In some embodiments, the model development module adjusts the hidden layer dimension d of the model based on robot dynamic parameters (such as joint degrees of freedom and mass matrix), generates a d×d dimensional orthogonal permutation matrix π based on a deep learning framework, and the public lightweight inference service model is a lightweight self-attention model, whose model parameter set f is... θ This includes parameters for the multi-head attention layer, the feedforward layer, and the layer normalization layer. Optionally, the model parameter set f can be adjusted based on the permutation matrix π using the model development module. θ The linkage parameters are transformed as follows:

[0080] Step 1: The parameters of the multi-head attention layer include the query matrix W. q Key matrix W k Value matrix W v and output matrix W o For the query matrix W q Key matrix W k Sum matrix W v Perform left multiplication of the permutation matrix transpose π respectively T Query matrix W q Replace with π T W q The key matrix W k Replace with π T W k The value matrix W v Replace with π T W v To maintain the correctness of attention calculation after input data permutation, the output matrix W is... o Perform right multiplication of the permutation matrix π T The output matrix W o Replace with W o π T To maintain consistency between the attention output and the original model;

[0081] Step 2: The feedforward layer parameters include the first linear layer parameter W1 and the second linear layer parameter W2. The first linear layer parameter W1 is multiplied by π on the left. T Replace W1 with π T W1, the second linear layer parameter W2, is right-multiplied by π. T Replace W2 with W2π T To maintain consistency between the nonlinear transformation and the original model;

[0082] Step 3: Layer Normalization. Layer parameters include scaling parameter γ and bias parameter β. Both scaling parameter γ and bias parameter β are multiplied by π on the right. TReplace the scaling parameter γ with γπ T Replace the bias parameter β with βπ T To maintain consistency between the standardized feature distribution and the original model.

[0083] By using the permutation matrix π, the model parameter set f θ Perform linkage parameter transformation to obtain the transformed f θ ′ f will be transmitted through public cloud nodes θ ′ The permutation matrix π is then sent to each robot.

[0084] In some embodiments, the transformation equivalence is preserved by the unitary property of the orthogonal permutation matrix π, as follows:

[0085] Forward transformation equivalence: The calculation result of the transformed model with the permutation input is equivalent to the calculation result of the original model with the permutation, maintaining the consistency of the reasoning logic;

[0086] Uniqueness of inverse transformation: Due to the orthogonality of the permutation matrix, the robot can recover the original reasoning result through inverse transformation without information loss;

[0087] Computational complexity is conserved: the computational complexity of parameter transformation and data substitution is low-order, which is negligible compared with the computational complexity of the original model inference, thus meeting the real-time requirements.

[0088] In some embodiments, a time-sensitive permutation strategy is introduced to address the timing characteristics of humanoid robot operations. This time-sensitive permutation strategy updates the permutation matrix and specifically includes:

[0089] Dynamic matrix update: The permutation matrix is ​​updated periodically with time steps. The update period (e.g., 500ms-2s) is adjusted according to the task type to prevent attackers from cracking the permutation pattern through time series feature analysis.

[0090] Cross-time step association: Maintaining motion continuity through orthogonal transformation relationship, so that the target robot can still maintain a smooth transition of the work process when the network is disconnected. In the application scenario of home robot, when the robot performs ironing work, the iron's movement trajectory can still be maintained after the joint angle sequence is replaced.

[0091] Interruption recovery mechanism: When the target robot goes offline, it caches the permutation matrix sequence of the most recent time step. After the network is restored, the time consistency of the job results is maintained by backtracking through the historical matrix.

[0092] In step S101 of some embodiments, optionally, multimodal data is dynamically acquired by using a multimodal sensor at the target robot end. The multimodal data includes at least visual image data, tactile signal data, and joint sequence data. The multimodal data is then transformed by matrix transformation according to a permutation matrix at the target robot end to generate multimodal transformation data.

[0093] Multimodal data x, including visual image data, tactile signal data, and joint sequence data, is collected from the target robot. The multimodal data x is then transformed using a locally stored permutation matrix π to obtain multimodal transformed data x. ′ = xπ.

[0094] For example, the target robot acquires a surface image of the work object through a camera (such as an RGB-D camera), extracts features to obtain a 256-dimensional visual feature vector x_vis, and executes x_vis. ′ =x_visπ; Collects tactile signals x_tact using a tactile sensor (such as a pressure sensor), and executes x_tact. ′ =x_tactπ; Collect the angle sequence x_joint of multiple joints of the robotic arm, and perform batch replacement x_joint. ′ = x_jointπ.

[0095] In some embodiments, step S102 may include, but is not limited to, steps S401 to S402.

[0096] Step S401: When the network communication status is connected, the multimodal transformation data is collaboratively reasoned and analyzed based on the data sensitivity of the multimodal transformation data through the public lightweight reasoning service model and the local reasoning service model of the target robot, and reasoning results are generated.

[0097] Step S402: When the network communication status is offline, the multimodal transformation data is inferred through the local inference service model of the target robot to generate inference results.

[0098] In some embodiments, each robot is equipped with a lightweight inference engine, which supports model deployment and computation in offline mode, enables local inference of multimodal transformation data, and avoids robot system stagnation caused by network interruption, resulting in production accidents or economic losses.

[0099] In some embodiments, step S401 may include, but is not limited to, steps S501 to S505.

[0100] Step S501: Perform data sensitivity detection on the multimodal transformation data and split the multimodal transformation data into sensitive data and non-sensitive data;

[0101] Step S502: The sensitive data is inferred through the local inference service model on the target robot to generate the first inference result;

[0102] Step S503: Through the target robot terminal, the non-sensitive data is encrypted and uploaded to the public cloud node. The encrypted non-sensitive data is decrypted through the public cloud node. The public lightweight inference service model is used to infer the decrypted non-sensitive data and generate a second inference result.

[0103] Step S504: The second inference result is encrypted and sent to the target robot terminal through the public cloud node. The target robot terminal decrypts the encrypted second inference result to obtain the decrypted second inference result.

[0104] Step S505: Determine the reasoning result based on the first reasoning result and the decrypted second reasoning result.

[0105] In step S501 of some embodiments, automatic labeling of data sensitivity levels is achieved through a multi-level classification model, which includes:

[0106] Feature extraction layer: CNN is used to extract visual features, and LSTM is used to extract temporal features of the task, forming a multimodal feature vector;

[0107] Label prediction layer: Based on semantic understanding of multimodal feature vectors, the model classifies multimodal transformation data into three levels: high sensitivity, medium sensitivity, and low sensitivity. High sensitivity data includes faces, medical data, core process parameters, etc., such as user activity videos captured by target humanoid robots in home service scenarios and assembly torque data of target robots in industrial scenarios. Medium sensitivity data includes joint angle sequences and work environment maps, such as limb movement data of robots during operation. Low sensitivity data includes environmental temperature and humidity, public weather data, etc., such as readings from ordinary sensors in a workshop.

[0108] Confidence calibration: The classification probability of multimodal transformation data is adjusted by confidence calibration related algorithms (such as logistic regression calibration Platt Scaling, etc.) to ensure that the mislabeling rate of highly sensitive data is less than a preset threshold (such as 0.5%), thus protecting privacy and security.

[0109] Multimodal transformation data is divided into sensitive data and insensitive data using a multi-level classification model. Sensitive data includes highly sensitive data in the multimodal transformation data, while insensitive data includes moderately sensitive data and low-sensitivity data in the multimodal transformation data.

[0110] In some embodiments, optionally, the x_vis data contained in the multimodal transformation data is processed through a local inference service model or a public lightweight inference service model on the target robot.′ x_tact ′ and x_joint ′ Joint reasoning is performed to calculate intermediate features. Intermediate features are abstract representations after multimodal transformation data fusion and are also intermediate states calculated within the model. They are used to describe the environmental state of the current operation and the deviation of robot actions. For example, in the household service scenario of ironing clothes, intermediate features may include, but are not limited to, the degree of wrinkles in the clothes, the distribution of iron pressure, and the deviation of the motion trajectory.

[0111] Using either the local inference service model or the public lightweight inference service model on the target robot, further inference is performed based on intermediate features to generate the final robot action execution command y. ′ Robot action execution command y ′ This is the final result of the reasoning.

[0112] In step S103 of some embodiments, the inference result is inversely transformed using the permutation matrix stored locally on the target robot to obtain motion control instructions; the target robot is then driven to perform corresponding actions according to the motion control instructions.

[0113] Optionally, the inference result y can be modified by the permutation matrix π. ′ Perform the inverse transformation to obtain the motion control command y = y ′ π T .

[0114] For example, taking the household service scenario of ironing clothes as an example, the action control instruction y is either "continue ironing" or "complete and disconnect power". The action control instruction y includes multiple action control dimensions, enabling the robot to complete the task accurately and efficiently. Taking the "continue ironing" instruction as an example, the "continue ironing" instruction covers multiple action control dimensions, specifically including:

[0115] 1) Path planning and movement commands: The target robot plans a reasonable movement path based on the shape and location of the area to be ironed and its current position. For example, when ironing flat clothes, a straight back-and-forth movement path is planned; if ironing objects with complex curved surfaces, the path planning is more flexible and varied, which involves the control of all the robot's motors, precisely adjusting the motor speed and angle, etc., to achieve precise movement of the robot on a two-dimensional plane;

[0116] 2) Robotic Arm Joint Angle Control Sequence: To ensure the iron accurately covers the area to be ironed and applies appropriate pressure, the robotic arm joints need to move in coordination, resulting in an extremely complex control sequence. Taking a robotic arm with multiple degrees of freedom as an example, the shoulder joint controls a wide range of horizontal and vertical movements, the elbow joint adjusts the arm's extension, and the wrist joint precisely controls the iron's posture. When approaching the area to be ironed, the shoulder joint first raises the arm at a specific angle, and the elbow joint bends slightly to position the iron at the appropriate height. After reaching the designated position, the wrist joint makes fine adjustments to ensure the bottom of the iron is parallel to the surface to be ironed. These joint angle controls need to be adjusted in real-time based on path planning and the surface conditions of the object, involving precise control of multiple joint motors. The rotation angle, speed, and acceleration of each joint motor must be strictly controlled to achieve smooth and precise movements.

[0117] 3) Ironing Pressure Control Command: To improve ironing results, the target robot adjusts the pressure applied to clothing with the iron in real time. Optionally, a force sensor detects the contact force between the iron and the clothing and feeds the signal back to the control system. When the detected pressure is too low, the control system issues a command to increase the downward pressure on the iron by increasing the rotation angle of the arm motor; conversely, when the pressure is too high, the downward pressure is reduced. For example, when ironing thin fabrics, the pressure is reduced to avoid damaging the clothing; when ironing thick fabrics, the pressure is increased so that the iron can effectively smooth out wrinkles.

[0118] 4) Motion Coordination and Rhythm Control Commands: The above-mentioned motion control commands are not executed in isolation, but require a high degree of coordination and control of the motion rhythm. For example, during robot movement, the movement of the arm joints must be synchronized with the movement of the chassis to ensure that the iron consistently and stably applies pressure and temperature to the area to be ironed; pressure and temperature adjustments must also match the target robot's motion rhythm to maintain accuracy in different movement phases and contact states. Simultaneously, the control of the motion rhythm must consider the uniformity of the ironing effect, avoiding uneven ironing caused by excessively fast or slow movements.

[0119] The following section provides a detailed description and explanation of the solutions in this embodiment of the invention, using specific examples of electromechanical equipment assembly in industrial manufacturing scenarios:

[0120] 1) Scenario Background: Industrial humanoid robots perform electromechanical equipment assembly tasks on an automotive parts production line, requiring them to simultaneously handle sensitive assembly process parameters (such as screw tightening torque) and non-sensitive material arrival information. When the production line network is interrupted due to a fault, the robot must continue to complete the current process and prevent process data from being leaked to the public network.

[0121] 2) Detailed implementation process:

[0122] Step 1: Perform data sensitivity classification and cloud routing for humanoid robots in industrial scenarios. Specifically, high-sensitivity data includes real-time torque values ​​(including process parameters) collected by torque sensors, which are marked as "high" and forcibly routed to the factory's private cloud; low-sensitivity data includes material supplier arrival time query requests, which are marked as "low" and routed to the public cloud to obtain real-time data.

[0123] Step 2: Collect multimodal data and perform transformation processing. Specifically, collect 16-bit torque sensor data x_torque (sampling at 100Hz) through the robot end, transform it into x_torque' = x_torqueπ through a permutation matrix, and transmit it to the private cloud via VPN encryption; transform the text-format material query request x_order into x_order' = x_orderπ and send it to the public cloud.

[0124] Step 3: Cross-cloud collaborative reasoning and result fusion for humanoid robots in industrial scenarios. Specifically, the local reasoning service model on the target robot performs transformed model reasoning on x_torque' to determine whether the torque meets the process requirements (e.g., 60±5Nm) and returns the transformed reasoning result y_torque'; the public reasoning service model on the public cloud node performs reasoning on x_order' to query the incoming material status and returns the transformed reasoning result y_order'.

[0125] Step 4: Using the permutation matrix, perform an inverse transformation on y_torque' and y_order' on the target robot to obtain y_torque = y_torque'π T And y_order = y_order'π T By combining y_torque and y_order, the conclusion "torque is qualified, material will arrive in 10 minutes" is obtained, and the action control instruction "stop the current process and wait for the material to arrive" is generated simultaneously.

[0126] 3) Emergency handling mechanism for network outage of humanoid robots in industrial scenarios: If the network outage occurs during the execution of a process: The robot calls the 10 most recent sets of process parameter models stored locally and performs rolling window inference on the current torque data to ensure assembly accuracy; the material arrival time is predicted by local cached historical data, with the error controlled within ±5 minutes to meet the production line buffer requirements; all process data during the network outage is temporarily stored on the local encrypted hard drive, and after the network is restored, it is synchronized to the private cloud through a secure channel to maintain data integrity.

[0127] Please see Figure 3 This application also provides a humanoid robot control device based on model reasoning, which can implement the above-described method. The device includes:

[0128] The first module is used to dynamically collect multimodal data through the target robot end, transform the multimodal data, and generate multimodal transformed data;

[0129] The second module is used to dynamically monitor the network communication status of the target robot. Based on the network communication status and the data sensitivity of the multimodal transformation data, it performs inference on the multimodal transformation data through a lightweight inference service model and outputs the inference results. The lightweight inference service model is obtained through full-layer parameter transformation and federated learning collaborative training.

[0130] The third module is used to perform inverse transformation on the reasoning results through the target robot to obtain motion control commands, and to control the target humanoid robot according to the motion control commands.

[0131] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0132] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0133] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0134] Please see Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0135] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0136] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the methods described in the embodiments of this application.

[0137] The input / output interface 903 is used to implement information input and output;

[0138] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0139] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0140] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0141] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0142] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0143] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0144] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0145] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0146] The humanoid robot control method, device, electronic device, storage medium, and program product based on model inference provided in this application dynamically collects multimodal data from the target robot, transforms the multimodal data to generate multimodal transformed data, and infers the multimodal transformed data through a lightweight inference service model based on the network communication status of the target robot and the data sensitivity of the multimodal transformed data, outputting the inference result. The target robot then performs an inverse transformation on the inference result to obtain motion control commands, and controls the target humanoid robot according to the motion control commands. This application can realize automated robot control based on multimodal data input. Through full-layer parameter transformation and federated learning collaborative training of the lightweight inference service model, it can meet the real-time control cycle requirement of 10-20ms without increasing the computational load on the robot. Based on the network communication status of the target robot and the data sensitivity of the multimodal data, model inference is performed, enabling the robot to maintain real-time task processing capabilities in network outage or weak network scenarios, avoiding operation interruption or safety accidents due to network interruption, while maintaining data security and improving robot control efficiency and accuracy.

[0147] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0148] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0151] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0152] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0154] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A humanoid robot control method based on model reasoning, characterized in that, The method includes the following steps: Multimodal data is dynamically collected from the target robot, and the multimodal data is transformed to generate multimodal transformed data. The network communication status of the target robot is dynamically monitored. Based on the network communication status and the data sensitivity of the multimodal transformation data, a lightweight inference service model is used to infer the multimodal transformation data and output the inference result. The lightweight inference service model is obtained through full-layer parameter transformation and federated learning collaborative training. The inference result is inversely transformed by the target robot to obtain motion control commands, and the target humanoid robot is controlled according to the motion control commands. The system dynamically monitors the network communication status of the target robot. Based on the network communication status and the data sensitivity of the multimodal transformation data, it performs inference on the multimodal transformation data using a lightweight inference service model and outputs the inference results, including: When the network communication status is connected, the multimodal transformation data is collaboratively reasoned and analyzed based on the data sensitivity of the multimodal transformation data through the public lightweight reasoning service model and the local reasoning service model of the target robot, and the reasoning result is generated. When the network communication status is offline, the multimodal transformation data is inferred through the local inference service model of the target robot to generate the inference result; When the network communication state is connected, the multimodal transformation data is collaboratively inferred and analyzed based on the data sensitivity of the multimodal transformation data, using the public lightweight inference service model and the local inference service model of the target robot, to generate the inference result, including: Data sensitivity detection is performed on the multimodal transformation data, and the multimodal transformation data is split into sensitive data and non-sensitive data; The sensitive data is inferred using the local inference service model on the target robot to generate a first inference result; The non-sensitive data is encrypted and uploaded to the public cloud node through the target robot terminal. The encrypted non-sensitive data is decrypted through the public cloud node. The decrypted non-sensitive data is then inferred using the public lightweight inference service model to generate a second inference result. The second inference result is encrypted and sent to the target robot terminal through the public cloud node, and the target robot terminal decrypts the encrypted second inference result to obtain the decrypted second inference result; The reasoning result is determined based on the first reasoning result and the decrypted second reasoning result.

2. The method according to claim 1, characterized in that, The method further includes: After performing full-layer parameter transformation on the public lightweight inference service model through public cloud nodes, the public lightweight inference service model and the permutation matrix are distributed to each robot terminal. Each robot receives a public task processing model and stores it locally as a corresponding local inference service model. The local dataset is used to train the local inference service model and generate corresponding model update parameters. Each robot terminal transmits the corresponding model update parameters, performs permutation and alignment of the model update parameters according to the permutation matrix, and encrypts and uploads the permutation and aligned model update parameters to the public cloud node. The public cloud node performs global model aggregation based on the encrypted model update parameters uploaded by each private cloud node, determines the aggregation result, and updates the public lightweight inference service model based on the aggregation result. The updated public lightweight inference service model is encrypted through the public cloud node and then distributed to each of the robot terminals. Each robot terminal updates its corresponding local inference service model based on the decrypted public lightweight inference service model.

3. The method according to claim 2, characterized in that, The step of dynamically acquiring multimodal data through the target robot end, transforming the multimodal data, and generating multimodal transformed data includes: The multimodal data is dynamically acquired through the target robot end using multimodal sensors. The multimodal data includes at least visual image data, tactile signal data, and joint sequence data. The target robot performs matrix transformation on the multimodal data according to the permutation matrix to generate the multimodal transformed data.

4. The method according to claim 2, characterized in that, The step of performing full-layer parameter transformation on the public lightweight inference service model through public cloud nodes, and then distributing the public lightweight inference service model and permutation matrix to each robot terminal includes: Obtain the model parameter set corresponding to the public lightweight inference service model. The model parameter set includes at least the multi-head attention layer parameters, the forward layer parameters, and the layer normalization layer parameters. The permutation matrix is ​​randomly generated, and the dimension of the permutation matrix is ​​consistent with the hidden layer dimension of the public lightweight inference service model. Using the permutation matrix, the model parameter set is subjected to linked parameter transformation to obtain a public lightweight inference service model after full-layer parameter transformation; The public lightweight inference service model after full-layer parameter transformation and the permutation matrix are distributed to each robot terminal through the public cloud node.

5. The method according to claim 2, characterized in that, The step of performing an inverse transformation on the reasoning result through the target robot to obtain motion control commands, and controlling the target humanoid robot according to the motion control commands, includes: The inference result is inversely transformed using the permutation matrix stored locally on the target robot to obtain the action control command; The target humanoid robot is driven to perform corresponding actions according to the motion control instructions via the target robot end.

6. A humanoid robot control device based on model reasoning, characterized in that, The device includes: The first module is used to dynamically collect multimodal data through the target robot end, transform the multimodal data, and generate multimodal transformed data; The second module is used to dynamically monitor the network communication status of the target robot. Based on the network communication status and the data sensitivity of the multimodal transformation data, it performs inference on the multimodal transformation data through a lightweight inference service model and outputs the inference result. The lightweight inference service model is obtained through full-layer parameter transformation and federated learning collaborative training. The third module is used to perform an inverse transformation on the reasoning result through the target robot end to obtain motion control instructions, and to control the target humanoid robot according to the motion control instructions; The system dynamically monitors the network communication status of the target robot. Based on the network communication status and the data sensitivity of the multimodal transformation data, it performs inference on the multimodal transformation data using a lightweight inference service model and outputs the inference results, including: When the network communication status is connected, the multimodal transformation data is collaboratively reasoned and analyzed based on the data sensitivity of the multimodal transformation data through the public lightweight reasoning service model and the local reasoning service model of the target robot, and the reasoning result is generated. When the network communication status is offline, the multimodal transformation data is inferred through the local inference service model of the target robot to generate the inference result; When the network communication state is connected, the multimodal transformation data is collaboratively inferred and analyzed based on the data sensitivity of the multimodal transformation data, using the public lightweight inference service model and the local inference service model of the target robot, to generate the inference result, including: Data sensitivity detection is performed on the multimodal transformation data, and the multimodal transformation data is split into sensitive data and non-sensitive data; The sensitive data is inferred using the local inference service model on the target robot to generate a first inference result; The non-sensitive data is encrypted and uploaded to the public cloud node through the target robot terminal. The encrypted non-sensitive data is decrypted through the public cloud node. The decrypted non-sensitive data is then inferred using the public lightweight inference service model to generate a second inference result. The second inference result is encrypted and sent to the target robot terminal through the public cloud node, and the target robot terminal decrypts the encrypted second inference result to obtain the decrypted second inference result; The reasoning result is determined based on the first reasoning result and the decrypted second reasoning result.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images