Model reasoning-based humanoid robot control method and related equipment

Abstract

The invention discloses a humanoid robot control method and related equipment based on model reasoning, and the method comprises the steps: dynamically collecting multi-modal data through a target robot end, converting the multi-modal data, and generating multi-modal conversion data; dynamically monitoring the network communication state of the target robot end, reasoning the multi-modal transformation data through a lightweight reasoning service model according to the network communication state and the data sensitivity of the multi-modal transformation data, and outputting a reasoning result; the lightweight reasoning service model is obtained through full-layer parameter transformation and federated learning cooperative training; and performing inverse transformation on the reasoning result through the target robot end to obtain an action control instruction, and controlling the target humanoid robot according to the action control instruction. According to the embodiment of the invention, robot automatic control based on multi-modal data input can be realized, and the robot control efficiency and accuracy are improved. The method can be widely applied to the technical field of robot control.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a humanoid robot control method and related equipment based on model reasoning. Background Art

[0002] Currently, target robots are increasingly used in industrial automation, home services, medical care and other fields. However, their technical implementation faces the dual challenges of real-time performance and data security. Existing solutions in related technologies have significant limitations:

[0003] 1) The target robot's motion control requires extremely high timeliness, with a typical motion control cycle of less than 10-20ms. However, traditional cloud-based inference-based AI model deployment methods have the following problems:

[0004] Uncontrollable network latency: Even though the theoretical latency of 5G networks is as low as 1ms, in real industrial environments, signal interference and bandwidth fluctuations can cause latency exceeding 50ms, causing robot movements to lag or even lose control.

[0005] The risk of network outages is fatal: In critical scenarios such as industrial production lines, network outages can cause robot systems to stagnate, leading to production accidents or economic losses.

[0006] 2) In traditional solutions, lightweight models (such as MobileNet) have difficulty handling complex multimodal fusion tasks of robots, and the risk of model stealing attacks still exists.

[0007] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0008] The main purpose of the embodiments of the present application is to propose a humanoid robot control method and related equipment based on model reasoning.

[0009] To achieve the above objectives, an embodiment of the present application provides a humanoid robot control method based on model reasoning, the method comprising the following steps:

[0010] Dynamically collecting multimodal data through a target robot end, transforming the multimodal data, and generating multimodal transformed data;

[0011] Dynamically monitor the network communication status of the target robot, and based on the network communication status and the data sensitivity of the multimodal transformation data, reason on the multimodal transformation data using a lightweight inference service model, and output an inference result; the lightweight inference service model is obtained through full-layer parameter transformation and federated learning collaborative training;

[0012] The target robot performs an inverse transformation on the inference result to obtain an action control instruction, and controls the target humanoid robot according to the action control instruction.

[0013] In some embodiments, the method further comprises:

[0014] After performing full-layer parameter transformation on the public lightweight inference service model through the public cloud node, the public lightweight inference service model and the permutation matrix are sent to each robot end;

[0015] Receiving the public task processing model through each of the robot terminals and storing it locally as a corresponding local reasoning service model, performing model training on the local reasoning service model using a local data set, and generating corresponding model update parameters;

[0016] Each robot terminal updates the corresponding model parameters, performs permutation and alignment on the model update parameters according to the permutation matrix, and encrypts and uploads the permuted and aligned model update parameters to the public cloud node. The public cloud node performs global model aggregation based on the encrypted model update parameters uploaded by each private cloud node, determines an aggregation result, and updates the public lightweight inference service model according to the aggregation result.

[0017] The updated public lightweight reasoning service model is encrypted by the public cloud node and sent to each of the robot terminals. Each of the robot terminals updates the corresponding local reasoning service model based on the decrypted public lightweight reasoning service model.

[0018] In some embodiments, dynamically collecting multimodal data through the target robot end, transforming the multimodal data, and generating multimodal transformation data include:

[0019] Dynamically collecting the multimodal data using a multimodal sensor through the target robot, wherein the multimodal data includes at least visual image data, tactile signal data, and joint sequence data;

[0020] The target robot performs matrix transformation on the multimodal data according to the permutation matrix to generate the multimodal transformation data.

[0021] In some embodiments, after performing full-layer parameter transformation on the public lightweight inference service model through the public cloud node, the public lightweight inference service model and the permutation matrix are sent to each robot terminal, including:

[0022] Obtain a model parameter set corresponding to the public lightweight inference service model, where the model parameter set includes at least multi-head attention layer parameters, forward layer parameters, and layer normalization layer parameters;

[0023] Randomly generating the permutation matrix, wherein the dimension of the permutation matrix is ​​consistent with the model hidden layer dimension of the public lightweight inference service model;

[0024] Using the permutation matrix, performing linkage parameter transformation on the model parameter set to obtain a public lightweight inference service model after full-layer parameter transformation;

[0025] The public lightweight inference service model and the permutation matrix after the full-layer parameter transformation are sent to each of the robot terminals through the public cloud node.

[0026] In some embodiments, the dynamically monitoring the network communication status of the target robot, reasoning the multimodal transformation data using a lightweight reasoning service model based on the network communication status and the data sensitivity of the multimodal transformation data, and outputting the reasoning result includes:

[0027] When the network communication state is a connected state, performing collaborative reasoning analysis on the multimodal transformation data according to the data sensitivity of the multimodal transformation data through the public lightweight reasoning service model and the local reasoning service model of the target robot to generate the reasoning result;

[0028] When the network communication state is an offline state, the multimodal transformation data is inferred through the local inference service model of the target robot to generate the inference result.

[0029] In some embodiments, when the network communication state is a connected state, performing collaborative reasoning analysis on the multimodal transformation data according to the data sensitivity of the multimodal transformation data through the public lightweight reasoning service model and the local reasoning service model of the target robot to generate the reasoning result includes:

[0030] Performing data sensitivity detection on the multimodal transformation data, and splitting the multimodal transformation data into sensitive data and non-sensitive data;

[0031] Reasoning the sensitive data using a local reasoning service model on the target robot to generate a first reasoning result;

[0032] The target robot end encrypts and uploads the non-sensitive data to the public cloud node, decrypts the encrypted non-sensitive data through the public cloud node, and uses the public lightweight reasoning service model to reason on the decrypted non-sensitive data to generate a second reasoning result;

[0033] Encrypting the second reasoning result and sending it to the target robot through the public cloud node, and decrypting the encrypted second reasoning result through the target robot to obtain a decrypted second reasoning result;

[0034] The reasoning result is determined according to the first reasoning result and the decrypted second reasoning result.

[0035] In some embodiments, the target robot performs an inverse transformation on the inference result to obtain an action control instruction, and controls the target humanoid robot according to the action control instruction, including:

[0036] Performing an inverse transformation on the inference result using the permutation matrix stored locally on the target robot to obtain the action control instruction;

[0037] The target humanoid robot is driven to perform corresponding actions according to the action control instructions through the target robot end.

[0038] To achieve the above objectives, another aspect of the present application provides a humanoid robot control device based on model reasoning, the device comprising:

[0039] The first module is used to dynamically collect multimodal data through the target robot end, transform the multimodal data, and generate multimodal transformation data;

[0040] The second module is configured to dynamically monitor the network communication status of the target robot, and based on the network communication status and the data sensitivity of the multimodal transformation data, infer the multimodal transformation data using a lightweight inference service model, and output an inference result; the lightweight inference service model is obtained through full-layer parameter transformation and federated learning collaborative training;

[0041] The third module is used to perform an inverse transformation on the inference result through the target robot end to obtain an action control instruction, and control the target humanoid robot according to the action control instruction.

[0042] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0043] To achieve the above objectives, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0044] The embodiments of the present application include at least the following beneficial effects: The present application provides a humanoid robot control method, device, electronic device and storage medium based on model reasoning, which dynamically collects multimodal data from a target robot end, transforms the multimodal data, generates multimodal transformation data, and infers the multimodal transformation data through a lightweight reasoning service model based on the network communication status of the target robot end and the data sensitivity of the multimodal transformation data, outputs the inference result, and inversely transforms the inference result through the target robot end to obtain action control instructions, and controls the target humanoid robot according to the action control instructions. The present application can realize robot automated control based on multimodal data input, and through full-layer parameter transformation of the lightweight reasoning service model and federated learning collaborative training, it can meet the real-time control cycle requirement of 10-20ms without increasing the computing load on the robot end. Based on the network communication status of the target robot end and the data sensitivity of the multimodal data, model reasoning is performed, so that the robot maintains real-time task processing capabilities in disconnected or weak network scenarios, avoids operation interruption or safety incidents caused by network interruption, maintains data security, and improves robot control efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of a humanoid robot control method based on model reasoning provided by an embodiment of the present application;

[0046] Figure 2 is another flow chart of a humanoid robot control method based on model reasoning provided by an embodiment of the present application;

[0047] Figure 3 1 is a schematic structural diagram of a humanoid robot control device based on model reasoning provided in an embodiment of the present application;

[0048] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0050] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0051] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" as used herein.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0053] The humanoid robot control method based on model reasoning provided in the embodiment of the present application relates to the field of information technology. The humanoid robot control method based on model reasoning provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the humanoid robot control method based on model reasoning, etc., but is not limited to the above forms.

[0054] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0055] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0056] Figure 1This is an optional flowchart of a humanoid robot control method based on model reasoning provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S103.

[0057] Step S101, dynamically collecting multimodal data through the target robot end, transforming the multimodal data, and generating multimodal transformation data;

[0058] Step S102: Dynamically monitor the network communication status of the target robot. Based on the network communication status and the data sensitivity of the multimodal transformation data, the lightweight inference service model is used to infer the multimodal transformation data and output the inference results. The lightweight inference service model is obtained through full-layer parameter transformation and federated learning collaborative training.

[0059] Step S103: Inversely transform the inference result through the target robot end to obtain the action control instruction, and control the target humanoid robot according to the action control instruction.

[0060] In some embodiments, reference Figure 2 The above method may further include steps S201 to S204.

[0061] Step S201: After performing full-layer parameter transformation on the public lightweight inference service model through the public cloud node, the public lightweight inference service model and the permutation matrix are sent to each robot terminal;

[0062] Step S202: Receive the public task processing model through each robot terminal and store it locally as a corresponding local reasoning service model. Use the local data set to train the local reasoning service model and generate corresponding model update parameters.

[0063] In step S203, each robot terminal updates the corresponding model parameters, aligns the model update parameters according to the permutation matrix, and encrypts and uploads the permuted and aligned model update parameters to the public cloud node. The public cloud node aggregates the global model based on the encrypted model update parameters uploaded by each private cloud node, determines the aggregation result, and updates the public lightweight inference service model based on the aggregation result.

[0064] In step S204, the updated public lightweight reasoning service model is encrypted through the public cloud node and sent to each robot terminal. Each robot terminal updates the corresponding local reasoning service model based on the decrypted public lightweight reasoning service model.

[0065] In some embodiments, each robot terminal is based on a high-performance computing server to meet the needs of localized processing of sensitive data, and the public cloud node is based on an elastic cloud computing platform to provide computing power acceleration for non-sensitive data.

[0066] Optionally, the public cloud node includes a model development module, which uses a combination of knowledge distillation and quantization compression technology, as follows:

[0067] 1) Knowledge Distillation: This technology migrates the knowledge of the original large model to a lightweight architecture. Through teacher-student network training, it retains core reasoning capabilities while reducing the number of model parameters by over 80%. In one implementation, a visual inspection model was compressed from 86M parameters to 17M, maintaining efficient operation on embedded devices.

[0068] 2) Mixed-precision quantization: Quantizes model weights from FP32 to FP16 or INT8, further reducing storage size by over 50%. The quantized model can be stored in the robot's embedded memory, adapting to mainstream storage capacities.

[0069] 3) Dynamic weight pruning: By regularizing and eliminating redundant connections, the model computational complexity can be reduced by 30%-40% while maintaining accuracy, significantly reducing the time required for matrix operations.

[0070] The robot is equipped with a heterogeneous computing architecture and achieves low-latency inference through three layers of optimization, as follows:

[0071] Hardware acceleration layer: Integrates SIMD instruction sets or dedicated matrix operation units to control the calculation delay of permutation matrix multiplication to less than 1ms, ensuring real-time performance.

[0072] Algorithm optimization layer: Using block matrix decomposition technology, large matrix operations are split into multiple sub-blocks for parallel calculation, and hardware thread parallel acceleration is used to improve computing efficiency.

[0073] Memory management layer: Design a hierarchical caching mechanism to store frequently accessed permutation matrices and intermediate features in the cache, reducing memory access latency and improving data reading speed.

[0074] In some embodiments, step S201 may include but is not limited to steps S301 to S304.

[0075] Step S301: Obtain a model parameter set corresponding to a public lightweight inference service model, where the model parameter set includes at least multi-head attention layer parameters, forward layer parameters, and layer normalization layer parameters;

[0076] Step S302: randomly generate a permutation matrix, wherein the dimension of the permutation matrix is ​​consistent with the model hidden layer dimension of the public lightweight inference service model;

[0077] Step S303: Using the permutation matrix, perform linkage parameter transformation on the model parameter set to obtain a public lightweight inference service model after full-layer parameter transformation;

[0078] In step S304, the public lightweight inference service model and permutation matrix after the full-layer parameter transformation are sent to each robot terminal through the public cloud node.

[0079] In some embodiments, the model development module adjusts the model hidden dimension d according to the robot dynamic parameters (such as joint degrees of freedom, mass matrix), generates a d×d dimensional orthogonal permutation matrix π based on the deep learning framework, and the public lightweight inference service model is a lightweight self-attention model. The model parameter set f of the lightweight self-attention model θ Including multi-head attention layer parameters, forward layer parameters and layer normalization layer parameters. Optionally, through the model development module, according to the permutation matrix π, the model parameter set f θ Perform linkage parameter transformation as follows:

[0080] Step 1: The multi-head attention layer parameters include the query matrix W q , bond matrix W k , value matrix W v And the output matrix W o , for the query matrix W q , bond matrix W k Sum matrix W v Left-multiply the permutation matrix transpose π T , the query matrix W q Replace with π T W q , the bond matrix W k Replace with π T W k , the value matrix W v Replace with π T W v , maintain the correctness of attention calculation after input data permutation, and the output matrix W o Right multiply the permutation matrix π T , the output matrix W o Replaced with W o π T , keep the attention output consistent with the original model;

[0081] Step 2: The forward layer parameters include the first linear layer parameter W1 and the second linear layer parameter W2. The first linear layer parameter W1 is multiplied by π on the left. T , replace W1 with π T W1, the second linear layer parameter W2 is multiplied by π on the right T , replace W2 with W2π T , keep the consistency of nonlinear transformation and original model;

[0082] Step 3: Layer normalization layer parameters include scaling parameter γ and bias parameter β, scaling parameter γ and bias parameter β are right multiplied by π T, replacing the scaling parameter γ with γπ T , replace the bias parameter β with βπ T , keeping the normalized feature distribution consistent with the original model.

[0083] By permuting the matrix π, the model parameter set f θ Perform linkage parameter transformation to obtain the transformed f θ ′ , through the public cloud node to f θ ′ The permutation matrix π is sent to each robot end.

[0084] In some embodiments, the transformation equivalence is maintained by the unitary property of the orthogonal permutation matrix π, as follows:

[0085] Forward transformation equivalence: The calculation result of the transformed model for the permuted input is equivalent to the permutation of the calculation result of the original model, maintaining the consistency of the reasoning logic;

[0086] Uniqueness of inverse transformation: Due to the orthogonality of the permutation matrix, the robot can restore the original reasoning result through inverse transformation without information loss;

[0087] Conservation of computational complexity: The computational complexity of parameter transformation and data replacement is low-order, which is negligible compared to the computational complexity of original model inference, meeting real-time requirements.

[0088] In some embodiments, a time-sensitive replacement strategy is introduced to address the timing characteristics of humanoid robots. The replacement matrix is ​​updated using the time-sensitive replacement strategy. The time-sensitive replacement strategy specifically includes:

[0089] Dynamic matrix update: The permutation matrix is ​​updated periodically with each time step. The update period (e.g., 500ms-2s) is adjusted according to the task type to prevent attackers from cracking the permutation pattern through timing feature analysis.

[0090] Cross-timestep association: Orthogonal transformation relationships are used to maintain motion continuity, allowing the target robot to maintain smooth transitions in its work process even when the network is disconnected. For example, in a household robot application scenario, when the robot is performing ironing operations, the continuity of the iron's movement trajectory can be maintained after the joint angle sequence is permuted.

[0091] Interruption recovery mechanism: When the target robot is offline, it caches the permutation matrix sequence of the most recent time step. After the network is restored, it backtracks through the historical matrix to maintain the temporal consistency of the job results.

[0092] In step S101 of some embodiments, optionally, multimodal data is dynamically collected through a target robot end using a multimodal sensor, and the multimodal data includes at least visual image data, tactile signal data, and joint sequence data; through the target robot end, the multimodal data is matrix transformed according to a permutation matrix to generate multimodal transformation data.

[0093] The target robot collects multimodal data x, including visual image data, tactile signal data, and joint sequence data, and transforms the multimodal data x through the locally stored permutation matrix π to obtain the multimodal transformation data x ′ =xπ.

[0094] For example, the target robot collects the surface image of the work object through a camera (such as an RGB-D camera, etc.), obtains a 256-dimensional visual feature vector x_vis through feature extraction, and executes x_vis ′ =x_visπ; collect tactile signals x_tact through tactile sensors (such as pressure sensors, etc.), and execute x_tact ′ =x_tactπ; collect the angle sequence x_joint of multiple joints of the robot arm and perform batch replacement x_joint ′ =x_jointπ.

[0095] In some embodiments, step S102 may include but is not limited to steps S401 to S402.

[0096] Step S401: When the network communication state is connected, a public lightweight reasoning service model and a local reasoning service model of the target robot are used to perform collaborative reasoning analysis on the multimodal transformation data according to the data sensitivity of the multimodal transformation data to generate a reasoning result.

[0097] Step S402: When the network communication state is offline, the local reasoning service model of the target robot is used to reason on the multimodal transformation data to generate a reasoning result.

[0098] In some embodiments, each robot is equipped with a lightweight inference engine to support model deployment and calculation in an offline state, realize local inference of multimodal transformation data, and avoid network interruption causing stagnation of the robot system, resulting in production accidents or economic losses.

[0099] In some embodiments, step S401 may include but is not limited to steps S501 to S505.

[0100] Step S501: performing data sensitivity detection on the multimodal transformation data, and splitting the multimodal transformation data into sensitive data and non-sensitive data;

[0101] Step S502: reasoning the sensitive data using the local reasoning service model of the target robot to generate a first reasoning result;

[0102] Step S503: The target robot end encrypts and uploads the non-sensitive data to the public cloud node, decrypts the encrypted non-sensitive data through the public cloud node, and uses the public lightweight reasoning service model to reason on the decrypted non-sensitive data to generate a second reasoning result;

[0103] Step S504: encrypt the second reasoning result and send it to the target robot via the public cloud node. The target robot decrypts the encrypted second reasoning result to obtain a decrypted second reasoning result.

[0104] Step S505: Determine the reasoning result according to the first reasoning result and the decrypted second reasoning result.

[0105] In step S501 of some embodiments, automatic labeling of data sensitivity levels is achieved through a multi-level classification model, which includes:

[0106] Feature extraction layer: CNN is used to extract visual features, and LSTM is used to extract task timing features to form a multimodal feature vector.

[0107] Label prediction layer: Based on the semantic understanding of multimodal feature vectors, the model classifies multimodal transformation data into three levels of sensitivity: high sensitivity, medium sensitivity, and low sensitivity. Highly sensitive data includes faces, medical data, and core process parameters, such as user activity videos captured by target humanoid robots in home service scenarios and assembly torque data of target robots in industrial scenarios. Medium-sensitive data includes joint angle sequences and work environment maps, such as limb movement data during robot operation. Low-sensitive data includes ambient temperature and humidity, public weather data, and other data, such as common sensor readings in workshops.

[0108] Confidence calibration: Confidence calibration-related algorithms (such as logistic regression calibration Platt Scaling) are used to adjust the classification probability of multimodal transformation data so that the mislabeling rate of highly sensitive data is less than a preset threshold (such as 0.5%), ensuring privacy and security.

[0109] The multimodal transformation data is split into sensitive data and non-sensitive data through a multi-level classification model, wherein the sensitive data includes the highly sensitive data in the multimodal transformation data, and the non-sensitive data includes the medium-sensitive data and the low-sensitive data in the multimodal transformation data.

[0110] In some embodiments, the x_vis contained in the multimodal transformation data is optionally processed by the local reasoning service model or the public lightweight reasoning service model of the target robot.′ 、x_tact ′ and x_joint ′ Perform joint reasoning and calculate intermediate features. Intermediate features are abstract representations of multimodal transformation data fusion and are also the intermediate states of internal model calculations. They are used to describe the environmental state of the current operation and the robot's motion deviation. For example, in the household service scenario of ironing clothes, intermediate features may include but are not limited to the degree of clothing wrinkles, iron pressure distribution, and motion trajectory deviation.

[0111] Through the local reasoning service model or public lightweight reasoning service model on the target robot side, further reasoning is performed based on the intermediate features to generate the final robot action execution instruction y ′ , robot action execution instruction y ′ This is the final inference result.

[0112] In step S103 of some embodiments, the inference result is inversely transformed through the permutation matrix stored locally on the target robot side to obtain the action control instruction; and the target humanoid robot is driven to perform the corresponding action according to the action control instruction through the target robot side.

[0113] Optionally, by permuting the matrix π, the inference result y ′ Perform the inverse transformation to obtain the action control instruction y=y ′ π T .

[0114] For example, in the household service scenario of ironing clothes, the action control instruction y is a "continue ironing" or "complete power off" instruction. The action control instruction y includes multiple action control dimension instructions, enabling the robot to complete the task accurately and efficiently. Taking the "continue ironing" instruction as an example, the "continue ironing" instruction covers multiple action control dimensions, including:

[0115] 1) Path Planning and Movement Instructions: The target robot plans a reasonable movement path based on the shape and position of the area to be ironed, as well as its current position. For example, when ironing flat clothing, a straight reciprocating movement path is planned. When ironing objects with complex curved surfaces, path planning is more flexible and involves controlling the robot's entire body motors, precisely adjusting motor speed and angle, etc., to achieve precise movement on a two-dimensional plane.

[0116] 2) Robotic arm joint angle control sequence: In order for the iron to accurately cover the area to be ironed and apply appropriate pressure, the robot arm joints need to move in coordination, and the control sequence is extremely complex. Taking a robotic arm with multiple degrees of freedom as an example, the shoulder joint needs to control the arm's large-scale movement in the horizontal and vertical directions, the elbow joint is responsible for adjusting the degree of arm extension, and the wrist joint is used to precisely control the posture of the iron. When approaching the area to be ironed, the shoulder joint first raises the arm at a specific angle, and the elbow joint bends moderately to keep the iron at an appropriate height; after reaching the designated position, the wrist joint is fine-tuned to ensure that the bottom surface of the iron is parallel to the surface to be ironed. The control of these joint angles needs to be adjusted in real time based on path planning and the surface conditions of the object, which involves the precise control of multiple joint motors. The rotation angle, speed, and acceleration of each joint motor must be strictly controlled to achieve smooth and precise movements;

[0117] 3) Iron pressure control instructions: To improve the ironing effect, the target robot adjusts the pressure applied to the clothes by the target robot using the iron in real time. Optionally, the contact force between the iron and the clothes is sensed by a force sensor, and the signal is fed back to the control system. When it detects that the pressure is too low, the control system will issue a command to increase the downward pressure on the iron by increasing the rotation angle of the arm motor; conversely, when the pressure is too high, the downward pressure is reduced. For example, when ironing light fabrics, the pressure is reduced to avoid damaging the clothes; when ironing heavy fabrics, the pressure is increased so that the iron can effectively smooth out wrinkles.

[0118] 4) Motion coordination and rhythm control instructions: The various motion control dimension instructions mentioned above are not executed in isolation; instead, they require a high degree of coordination and controlled rhythm. For example, during robot movement, the movement of the arm joints must be synchronized with the movement of the chassis, ensuring that the iron always and stably acts on the area to be ironed. Pressure and temperature regulation must also match the target robot's motion rhythm, maintaining pressure and temperature accuracy during different movement phases and contact states. Furthermore, the control of the motion rhythm must take into account the uniformity of the ironing effect, avoiding uneven ironing due to excessively fast or slow movements.

[0119] The following describes and explains the solution of the embodiment of the present invention in detail with reference to an example of electromechanical equipment assembly in a specific industrial manufacturing scenario.

[0120] 1) Scenario Background: Industrial humanoid robots assemble electromechanical equipment on an automotive parts production line, processing both sensitive assembly process parameters (such as screw tightening torque) and non-sensitive incoming material information. If the production line network is interrupted due to a fault, the robot must continue to complete the current process while preventing process data from being leaked to the public network.

[0121] 2) Detailed implementation process:

[0122] Step 1: Perform data sensitivity classification and cloud routing for industrial humanoid robots. Specifically, highly sensitive data, including real-time torque values ​​(including process parameters) collected by the torque sensor, is marked as "high" and forcibly routed to the factory's private cloud. Low-sensitivity data, including requests to query material supplier arrival times, is marked as "low" and routed to the public cloud for real-time data.

[0123] Step 2: Collect multimodal data and transform it. Specifically, the robot collects 16-bit torque sensor data x_torque (100Hz sampling), transforms it into x_torque'=x_torqueπ through a permutation matrix, and transmits it to the private cloud via VPN encryption. The incoming material query request x_order in text format is transformed into x_order'=x_orderπ and then sent to the public cloud.

[0124] Step 3: Cross-cloud collaborative reasoning and result fusion of the industrial humanoid robot. Specifically, the local reasoning service model on the target robot performs transformed model reasoning on x_torque' to determine whether the torque meets the process requirements (e.g., 60±5Nm), and returns the transformed reasoning result y_torque'. The public reasoning service model on the public cloud node performs reasoning on x_order' to query the incoming material status and returns the transformed reasoning result y_order'.

[0125] Step 4: Perform an inverse transformation on y_torque' and y_order' based on the permutation matrix on the target robot to obtain y_torque = y_torque'π T and y_order = y_order'π T , integrating y_torque and y_order to obtain the conclusion that "the torque is qualified and the material will arrive in 10 minutes", and simultaneously generating the action control instruction of "stop the current process and wait for the material to arrive";

[0126] 3) Emergency response mechanism for humanoid robots in industrial scenarios when the network is disconnected: If a network interruption occurs during process execution: the robot side calls the 10 most recent sets of process parameter models stored locally, and performs rolling window reasoning on the current torque data to ensure assembly accuracy; the incoming material time is predicted through locally cached historical data, and the error is controlled within ±5 minutes to meet the production line buffer requirements; all process data during the network outage is temporarily stored on a local encrypted hard drive, and after the network is restored, it is synchronized to the private cloud through a secure channel to maintain data integrity.

[0127] See also Figure 3 The present application also provides a humanoid robot control device based on model reasoning, which can implement the above method. The device includes:

[0128] The first module is used to dynamically collect multimodal data through the target robot end, transform the multimodal data, and generate multimodal transformation data;

[0129] The second module is used to dynamically monitor the network communication status of the target robot. Based on the network communication status and the data sensitivity of the multimodal transformation data, it uses a lightweight inference service model to infer the multimodal transformation data and output the inference results. The lightweight inference service model is obtained through full-layer parameter transformation and federated learning collaborative training.

[0130] The third module is used to perform inverse transformation on the inference result through the target robot end, obtain action control instructions, and control the target humanoid robot according to the action control instructions.

[0131] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0132] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0133] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0134] See also Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0135] The processor 901 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0136] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the above-mentioned methods of the embodiments of this application.

[0137] Input / output interface 903, used to implement information input and output;

[0138] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0139] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0140] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0141] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above method is implemented.

[0142] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0143] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0144] It is understandable that the contents of the above method embodiments are all applicable to the present program product embodiments, the functions specifically implemented by the present program product embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0145] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The present invention provides a humanoid robot control method, device, electronic device, storage medium, and program product based on model reasoning. The method dynamically collects multimodal data from a target robot, transforms the multimodal data, generates multimodal transformation data, and infers the multimodal transformation data based on the network communication status of the target robot and the data sensitivity of the multimodal transformation data through a lightweight reasoning service model, outputs the inference result, and inversely transforms the inference result through the target robot to obtain motion control instructions, and controls the target humanoid robot according to the motion control instructions. The present invention can realize robot automation control based on multimodal data input. By transforming the full-layer parameters of the lightweight reasoning service model and conducting federated learning collaborative training, the method can meet the real-time control cycle requirement of 10-20ms without increasing the computing load on the robot. Based on the network communication status of the target robot and the data sensitivity of the multimodal data, the method performs model reasoning, so that the robot can maintain real-time task processing capabilities in scenarios with disconnected or weak networks, avoids operation interruptions or safety accidents caused by network interruptions, maintains data security, and improves robot control efficiency and accuracy.

[0147] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0148] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0150] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0151] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0152] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0154] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0156] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0157] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A humanoid robot control method based on model reasoning, characterized in that: The method comprises the following steps: Dynamically collecting multimodal data through a target robot end, transforming the multimodal data, and generating multimodal transformed data; Dynamically monitor the network communication status of the target robot, and based on the network communication status and the data sensitivity of the multimodal transformation data, reason on the multimodal transformation data using a lightweight inference service model, and output an inference result; the lightweight inference service model is obtained through full-layer parameter transformation and federated learning collaborative training; The target robot performs an inverse transformation on the inference result to obtain an action control instruction, and controls the target humanoid robot according to the action control instruction.

2. The method according to claim 1, characterized in that The method further comprises: After performing full-layer parameter transformation on the public lightweight inference service model through the public cloud node, the public lightweight inference service model and the permutation matrix are sent to each robot end; Receiving the public task processing model through each of the robot terminals and storing it locally as a corresponding local reasoning service model, performing model training on the local reasoning service model using a local data set, and generating corresponding model update parameters; Each robot terminal updates the corresponding model parameters, performs permutation and alignment on the model update parameters according to the permutation matrix, and encrypts and uploads the permuted and aligned model update parameters to the public cloud node. The public cloud node performs global model aggregation based on the encrypted model update parameters uploaded by each private cloud node, determines an aggregation result, and updates the public lightweight inference service model according to the aggregation result. The updated public lightweight reasoning service model is encrypted by the public cloud node and sent to each of the robot terminals. Each of the robot terminals updates the corresponding local reasoning service model based on the decrypted public lightweight reasoning service model.

3. The method according to claim 2, characterized in that The method of dynamically collecting multimodal data through the target robot end and transforming the multimodal data to generate multimodal transformed data includes: Dynamically collecting the multimodal data using a multimodal sensor through the target robot, wherein the multimodal data includes at least visual image data, tactile signal data, and joint sequence data; The target robot performs matrix transformation on the multimodal data according to the permutation matrix to generate the multimodal transformation data.

4. The method according to claim 2, characterized in that After performing full-layer parameter transformation on the public lightweight reasoning service model through the public cloud node, the public lightweight reasoning service model and the permutation matrix are sent to each robot terminal, including: Obtain a model parameter set corresponding to the public lightweight inference service model, where the model parameter set includes at least multi-head attention layer parameters, forward layer parameters, and layer normalization layer parameters; Randomly generating the permutation matrix, where the dimension of the permutation matrix is ​​consistent with the model hidden layer dimension of the public lightweight inference service model; Using the permutation matrix, performing linkage parameter transformation on the model parameter set to obtain a public lightweight inference service model after full-layer parameter transformation; The public lightweight inference service model and the permutation matrix after the full-layer parameter transformation are sent to each of the robot terminals through the public cloud node.

5. The method according to claim 2, characterized in that The dynamically monitoring the network communication status of the target robot end, reasoning the multimodal transformation data through a lightweight reasoning service model according to the network communication status and the data sensitivity of the multimodal transformation data, and outputting the reasoning result, includes: When the network communication state is a connected state, performing collaborative reasoning analysis on the multimodal transformation data according to the data sensitivity of the multimodal transformation data through the public lightweight reasoning service model and the local reasoning service model of the target robot to generate the reasoning result; When the network communication state is an offline state, the multimodal transformation data is inferred through the local inference service model of the target robot to generate the inference result.

6. The method according to claim 5, characterized in that When the network communication state is a connected state, performing collaborative reasoning analysis on the multimodal transformation data according to the data sensitivity of the multimodal transformation data through the public lightweight reasoning service model and the local reasoning service model of the target robot to generate the reasoning result includes: Performing data sensitivity detection on the multimodal transformation data, and splitting the multimodal transformation data into sensitive data and non-sensitive data; Reasoning the sensitive data using a local reasoning service model on the target robot to generate a first reasoning result; The target robot end encrypts and uploads the non-sensitive data to the public cloud node, decrypts the encrypted non-sensitive data through the public cloud node, and uses the public lightweight reasoning service model to reason on the decrypted non-sensitive data to generate a second reasoning result; Encrypting the second reasoning result and sending it to the target robot through the public cloud node, and decrypting the encrypted second reasoning result through the target robot to obtain a decrypted second reasoning result; The reasoning result is determined according to the first reasoning result and the decrypted second reasoning result.

7. The method according to claim 2, characterized in that The inverse transformation of the inference result by the target robot end to obtain an action control instruction, and controlling the target humanoid robot according to the action control instruction includes: Performing an inverse transformation on the inference result using the permutation matrix stored locally on the target robot to obtain the action control instruction; The target humanoid robot is driven to perform corresponding actions according to the action control instructions through the target robot end.

8. A humanoid robot control device based on model reasoning, characterized in that: The device comprises: The first module is used to dynamically collect multimodal data through the target robot end, transform the multimodal data, and generate multimodal transformation data; The second module is configured to dynamically monitor the network communication status of the target robot, and based on the network communication status and the data sensitivity of the multimodal transformation data, infer the multimodal transformation data using a lightweight inference service model, and output an inference result; the lightweight inference service model is obtained through full-layer parameter transformation and federated learning collaborative training; The third module is used to perform an inverse transformation on the inference result through the target robot end to obtain an action control instruction, and control the target humanoid robot according to the action control instruction.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images